{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2020/21\n",
    "\n",
    "### Practice - Collaborative Boosted FW\n",
    "\n",
    "\n",
    "### Underlying idea... the way to capture the feature importance from the user point of view, is to... use their point of view (e.g., ratings, collaborative similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A few info about feature weightign techniques\n",
    "\n",
    "* Information retrieval methods (TF-IDF, BM25..) do not optimize a predictive model\n",
    "* Embedding methods (FISM, UFSM, Factorization Machines) optimize everything at the same time, model and weights, but are difficult to train and sensitive to noise\n",
    "* Wrapper methods (LFW, CFW) apply a two step approach approximating an already built collaborative model, more robust and easier to tune\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens10M: Verifying data consistency...\n",
      "Movielens10M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 10681\n",
      "\tNumber of users: 69878\n",
      "\tNumber of interactions in URM_all: 10000054\n",
      "\tValue range in URM_all: 0.50-5.00\n",
      "\tInteraction density: 1.34E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.43E+02\n",
      "\t\t Max: 7.36E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 9.36E+02\n",
      "\t\t Max: 3.49E+04\n",
      "\tGini Index: 0.57\n",
      "\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 20, feature occurrences: 21564, density 1.01E-01\n",
      "\tICM name: ICM_tags, Value range: 1.00 / 69.00, Num features: 10217, feature occurrences: 108563, density 9.95E-04\n",
      "\tICM name: ICM_all, Value range: 1.00 / 69.00, Num features: 10237, feature occurrences: 130127, density 1.19E-03\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from Notebooks_utils.data_splitter import train_test_holdout\n",
    "from Data_manager.Movielens.Movielens10MReader import Movielens10MReader\n",
    "\n",
    "data_reader = Movielens10MReader()\n",
    "data_loaded = data_reader.load_data()\n",
    "\n",
    "URM_all = data_loaded.get_URM_all()\n",
    "ICM_tags = data_loaded.get_ICM_from_name(\"ICM_tags\")\n",
    "ICM_genres = data_loaded.get_ICM_from_name(\"ICM_genres\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "URM_train, URM_test = train_test_holdout(URM_all, train_perc = 0.8)\n",
    "URM_train, URM_validation = train_test_holdout(URM_train, train_perc = 0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To shorten the notebook let's use the parameters we computed the previous time (Warning: not very clean, the data split may be different, also, running BayesianSearch multiple times may result in a different solution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_params_ItemKNNCF = {'topK': 700, 'shrink': 200, 'similarity': 'cosine', 'normalize': True}\n",
    "best_params_ItemKNNCBF = {'topK': 500, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Create the two models, collaborative and content based"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ItemKNNCFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 899.83 column/sec, elapsed time 0.20 min\n",
      "ItemKNNCBFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 9965.86 column/sec, elapsed time 0.02 min\n"
     ]
    }
   ],
   "source": [
    "from KNN.ItemKNNCFRecommender import ItemKNNCFRecommender\n",
    "from KNN.ItemKNNCBFRecommender import ItemKNNCBFRecommender\n",
    "\n",
    "\n",
    "itemKNNCF = ItemKNNCFRecommender(URM_train)\n",
    "itemKNNCF.fit(**best_params_ItemKNNCF)\n",
    "\n",
    "itemKNNCBF = ItemKNNCBFRecommender(URM_train, ICM_tags)\n",
    "itemKNNCBF.fit(**best_params_ItemKNNCBF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question, how the two similarities differ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_CF = itemKNNCF.W_sparse\n",
    "W_sparse_CBF = itemKNNCBF.W_sparse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's plt the global similarity distribution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_CF_sorted = np.sort(W_sparse_CF.data.copy())\n",
    "W_sparse_CBF_sorted = np.sort(W_sparse_CBF.data.copy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7.9525911e-05, 8.0648650e-05, 8.8569628e-05, ..., 5.9590876e-01,\n",
       "       6.0201591e-01, 6.0201591e-01], dtype=float32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W_sparse_CF_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5.2332488e-04, 5.8817206e-04, 5.9291383e-04, ..., 6.8691748e-01,\n",
       "       6.9236338e-01, 6.9236338e-01], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W_sparse_CBF_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8denafataZKuaWlpC4UCLRCKgsomDiCCKCjgMqJjh3mIDoPjjKPjqDM6D/w5OqigUBDBtSKIA1hFxYKKom2xQFsodKVpaZumS5p9+/z+OCfpbXqT3rQ5uffkvp+Px33ce8/6SZqez/ku5/s1d0dERLLXmHQHICIi6aVEICKS5ZQIRESynBKBiEiWUyIQEclyY9MdwFBVVVX5jBkz0h2GiEisrFy5cre7VydbF7tEMGPGDFasWJHuMEREYsXMtgy0TlVDIiJZTolARCTLKRGIiGS5SNsIzOwS4GtADnCPu9/ab/0ngPckxHISUO3ue4Zyns7OTurq6mhraxuGqDNLQUEBNTU15ObmpjsUERmlIksEZpYD3AFcDNQBy83sEXdf27uNu38Z+HK4/duAfxpqEgCoq6ujtLSUGTNmYGbD8wNkAHenoaGBuro6Zs6cme5wRGSUirJqaCGw3t03unsHsAS4cpDtrwN+dDQnamtro7KyclQlAQAzo7KyclSWdEQkc0SZCKYCWxO+14XLDmNmRcAlwEMDrF9kZivMbEV9fX3Sk422JNBrtP5cIpI5okwEya5gA415/Tbg6YGqhdx9sbvXunttdXXS5yFEBvfUl+GV36Q7CpGMFGUiqAOmJXyvAbYPsO21HGW1UCbZsWMH1157LbNmzeLkk0/msssu4+WXX6awsJAFCxb0vTo6OtIdavZ5+jbYuCzdUYhkpCh7DS0H5pjZTGAbwcX++v4bmVk5cB7w3ghjiZy7c9VVV/G3f/u3LFmyBIBVq1axc+dOZs2axapVq9IcoYhIcpElAnfvMrObgMcJuo/e6+5rzOzGcP2d4aZXAb9y9+aoYhkJy5YtIzc3lxtvvLFv2YIFC9i8eXP6ghIRSUGkzxG4+1Jgab9ld/b7fh9w33Cd8/OPrmHt9sbhOhwAJ08p47NvmzfoNqtXr+bMM89Mum7Dhg0sWLAAgHPPPZc77rhjWOMTETkWsRt0Lo5UNSQimWzUJYIj3blHZd68eTz44INpObeIyLHQWEPD5MILL6S9vZ277767b9ny5cvZsmXAkV9FRDKCEsEwMTMefvhhfv3rXzNr1izmzZvH5z73OaZMmZLu0EREBjXqqobSacqUKTzwwAOHLV+9enUaohERSY1KBCIiWU6JQEQkyykRiIhkOSUCEZEsp0QgIpLllAhERLKcEsEwOtIw1PPnz+ecc85h3bp1ADz55JOUl5f3DU/95je/Oc0/gYhkIz1HMExSHYb6rrvu4r//+7+5//77AXjjG9/IY489lra4RURUIhgmAw1DPW3atEO2a2xspKKiYqTDExEZ0OgrEfzik7DjheE95qRT4dJbB90klWGoDxw4QEtLC3/+85/71v3+97/vG6L6mmuu4dOf/vTwxS0ikoLRlwgyUGLV0I9//GMWLVrEL3/5S0BVQyKSfqMvERzhzj0qqQ5DfcUVV3DDDTeMQEQiIqlRG8EwSXUY6j/84Q/MmjVrpMMTERnQ6CsRpEnvMNQ333wzt956KwUFBcyYMYPbbrutr43A3cnLy+Oee+5Jd7giIn0iTQRmdgnwNYLJ6+9x98PqbczsfOA2IBfY7e7nRRlTlAYahrq1tTXp9ueffz7nn39+xFGJiAwuskRgZjnAHcDFQB2w3Mwecfe1CduMA74JXOLur5rZhKjiERGR5KJsI1gIrHf3je7eASwBruy3zfXAT939VQB33xVhPCIisfXh767ggRVbIzl2lIlgKpAYdV24LNEJQIWZPWlmK83s/Ud7Mnc/2l0z2mj9uURkaJ5aV8/G+uZIjh1lG4ElWdb/qjYWOBO4CCgE/mRmz7j7y4ccyGwRsAhg+vTphx20oKCAhoYGKisrMUt22nhydxoaGigoKEh3KCKSZo4T1eUtykRQBySOr1ADbE+yzW53bwaazex3wHzgkETg7ouBxQC1tbWH3SLX1NRQV1dHfX39MIafGQoKCqipqUl3GCKSAaK6zY0yESwH5pjZTGAbcC1Bm0Ci/wNuN7OxQB5wNvC/Qz1Rbm4uM2fOPMZwRUQyV5S1xJElAnfvMrObgMcJuo/e6+5rzOzGcP2d7v6imf0SeB7oIehiujqqmERE4sohllVDuPtSYGm/ZXf2+/5l4MtRxiEiEnfujkVUOaQhJkREYiKqEoESgYhIDETZkVyJQEQkBtyj6zWkRCAiEhcR1Q0pEYiIxIRKBCIiWSrqoWaUCEREMlxvHlCvIRGRLNVbHtBzBCIiWaq3akglAhGRLKfGYhGRLBX1rCRKBCIiGU6NxSIiWc7pbSNQY7GIiERAiUBEJMNFPXW5EoGISEyojUBEJEv1NRbrgTIRkeymEoGISJbyiJ8kUCIQEclwB6uGohFpIjCzS8xsnZmtN7NPJll/vpntN7NV4es/ooxHsljU3S5EItQ36FxEmWBsNIcFM8sB7gAuBuqA5Wb2iLuv7bfp79398qjiEBGJu75B52LYWLwQWO/uG929A1gCXBnh+UQGoRKBxF8cG4unAlsTvteFy/p7vZk9Z2a/MLN5yQ5kZovMbIWZraivr48iVhnt3KP7XyQSsTgPOpfsf13/n+dZ4Dh3nw98A/hZsgO5+2J3r3X32urq6mEOU7KDE11Tm0i04vxkcR0wLeF7DbA9cQN3b3T3pvDzUiDXzKoijEmymUoEEld9o4/Gr41gOTDHzGaaWR5wLfBI4gZmNsnCn8zMFobxNEQYk2Qr9RqSUSCqW5nIeg25e5eZ3QQ8DuQA97r7GjO7MVx/J3A18A9m1gW0Ate663+sREFVQxJfUT9QFlkigL7qnqX9lt2Z8Pl24PYoYxAB1FgssaaJaUSGjRKBxFPfA2URHT+lRGBmx5nZm8PPhWZWGlE8IhFRjaPEX9oai83sw8CDwF3hohoG6OYpkrFUNSQxFnXTaSolgo8A5wKNYUCvABOiDEpk+KmxWOIr6rGGUkkE7eEQEWEgNhaVsyVuVCKQGMuE0UefMrNPAYVmdjHwE+DRiOIREZGBpPGBsk8C9cALwN8TdAf990iiEYmMqoYkvtL+HIG79wB3hy+R+FLVkMRVxFVDR0wEZraJJG0C7n58JBGJDDePuhe2SLQyYWKa2oTPBcA1wPhowhGJkEoEElMHG4vT1Ebg7g0Jr23ufhtwYSTRiERBw1fJKJG2EoGZnZHwdQxBCUFPFkuMqGpI4i3tjcXAVxI+dwGbgXdFEo1IFKIesUskYlE/R5BKr6ELIjq3yAhTIpB4SltjsZndMtiO7v7V4Q9HJApqI5DRIarG4sFKBGoHkNEh6nK1SMSiHnRuwETg7p+P9MwiI0aNxRJvUT8Kk0qvoQLgQ8A8gucIwsD8g9GEJBIRNRZLzKVz0LnvAZOAvwGeIpiP4EBE8YgMPz1HIKNE2iamAWa7+2eAZne/H3grcGok0YhEQlVDEm9R38ukkgg6w/d9ZnYKUA7MSOXgZnaJma0zs/Vm9slBtjvLzLrN7OpUjisyJHqOQGKu94GydFYNLTazCuAzwCPAWuBLR9rJzHKAO4BLgZOB68zs5AG2+xLw+BDiFjkKSgQST1Hfy6TyZPF33L2boH1gKCOOLgTWu/tGADNbAlxJkEgSfRR4CDhrCMcWGQK1EUi8ZcJUlZvMbLGZXWRDa6mYCmxN+F4XLutjZlOBq4A7BzuQmS0ysxVmtqK+vn4IIYigqiGRI0glEZwI/IZgEvvNZna7mb0hhf2S/a/rf2t2G/CvYYljQO6+2N1r3b22uro6hVOLJFJjscRb7wNl6XiyuDeAVuAB4IGwreBrBNVEOUfYtQ6YlvC9Btjeb5taYElY0KgCLjOzLnf/WWrhiwyBSgQSU5kwMQ1mdh7wboKG3+WkNvrocmCOmc0EtgHXAtcnbuDuMxPOcR/wmJKADDs9RyAxF/WfcKpTVa4iKBV8wt2bUzmwu3eZ2U0EvYFygHvdfY2Z3RiuH7RdQGT4qGpIRoeoHihLpUQw390bj+bg7r4UWNpvWdIE4O4fOJpziByRGosl9qItEqQyVeVRJQGRzKNEIPEU9QC6qfQaEok5tRFIvKX9OYLwyV+R+FLVkIwSUXUfTaVEsN7MvpxseAiReFEikHjKhEHnTgNeBu4xs2fCp3zLog1LZBipRCAx1zfoXLqqhtz9gLvf7e7nAP8CfBZ4zczuN7PZ0YQlEgUlAomntDcWm1mOmV1hZg8TPFX8FYLB5x6lX9dQkcykxmIZHdL5ZPErwDLgy+7+x4TlD5rZm6IJS2QYqWpIYq6rO/gbzhkTTUfPVBLB+939D4kLzOxcd3/a3T8WSVQiw0olAom31s5gXM7C3Gg6caaSXr6eZNk3hjsQkcipRCAx1ZcI8ka4RGBmrwfOAarN7JaEVWUceeRRkcyhQeck5trCRFAQUYlgsKqhPKAk3KY0YXkjoLmFJUY06JzEW9oSgbs/BTxlZve5+5ZIzi4yEtRYLDHXFnEbwWBVQ7e5+83A7WZ2WNna3a+IJCKRyCgRSDy1dqQpEQDfC9//J5Izi4wYtRFIvLV29gDpqRpaGQ4492F3f28kZxcZCaoakpjrrRrKHxtNr6FBjxpOKl9tZnmRnF1kRKixWOKtrbObgtwxjBmTvhnKNgNPm9kjQN80le7+1UgiEomKSgQSU62d3ZFVC0FqiWB7+BrDod1IReJBzxFIzLV1dkfWUAwpJAJ3//zRHtzMLiEYqC4HuMfdb+23/krgv4AeoAu4uf9wFiLHTlVDEm+tnT3pTQRmVk0w/PQ8oKB3ubtfeIT9coA7gIuBOmC5mT3i7msTNnsCeMTd3cxOAx4A5g75pxAZjBqLJebaOrvJjzARpNIE/QPgJWAm8HmCNoPlKey3EFjv7hvdvQNYAlyZuIG7N7n3lduLUT8/iZQSgcRTUDUU3RTzqRy50t2/DXS6+1Pu/kHgdSnsNxXYmvC9Llx2CDO7ysxeAn4OfDDZgcJZ0VaY2Yr6+voUTi2SSPcXEm9N7V0U56fSpHt0UkkEneH7a2b2VjM7HahJYb9kt1/JnlB+2N3nAm8naC84fCf3xe5e6+611dXVKZxaJIGqhiTm9jZ3MK4oul78qaSYL5hZOfBxguGny4B/SmG/OmBawvcagt5HSbn778xslplVufvuFI4vkiI1Fku87WnuoKIoN7Ljp9Jr6LHw437ggiEcezkwx8xmAtuAa4HrEzcI5zzeEDYWn0Ew4mnDEM4hkjqVCCSGOrp6aGzrorI4P7JzDDbo3DcYpHL1SLOTuXuXmd0EPE7QffRed19jZjeG6+8E3gm838w6gVbg3QmNxyLDQ39SEmO7m9oBqC5NQyIAVhzrwd19Kf0muA8TQO/nLwFfOtbziKRGJQKJnx2NbQBMLi84wpZHb7BB5+6P7KwiI0mNxRJjr+0LEsGkdCSC3vkIzOxRkvf20XwEEhNqLJb42rq3BYCaisLIzqH5CEREMtiWhhbGFeVSWpCGXkPuvjJ8fyqys4uMBFUNSYxt3t3M8VXFkZ7jiA+UmdnlZvZXM9tjZo1mdsDMGiONSmRYqdeQxNfG3U3MiDgRpPJA2W3AO4AX1LVTYkklAomp/S2d7Gxs54SJ0c4AkMoQE1uB1UoCEn9KBBIva18LKl9OmlwW6XlSKRH8C7DUzJ4C2nsXaoYyiQ/dw0g8rdm+H4CTMyARfBFoIpiLQHMXS/yoakhiauWWvdRUFEb6VDGklgjGu/tbIo1CJFJ6jkDix91Zvnkvb5xTFfm5Umkj+I2ZKRFI/KlEIDHy0o4D7G5q5/WzKiM/VyqJ4CPAL82sVd1HJZbUz0FiaNm6XQCcd0L0c7CkMgx1tP2WRCKnqiGJn8dX72B+TTkTy6IbY6jXYGMNzXX3l8J5Ag7j7s9GF5bIMFJjscTMqw0tPFe3n3+9ZO6InG+wEsEtwCLgK0nWOXBhJBGJREaJQOLhwZVbMYMrFkwZkfMNNtbQovB9KLOSiWQgtRFIfHR09fDDv2zlghMnMHVcdCOOJhqwsdjMzjKzSQnf329m/2dmXzez8SMSnchwUNWQxMhjz29nd1M773v9cSN2zsF6Dd0FdACY2ZuAW4HvEsxdvDj60ESGixqLJR66e5w7lq1n7qRSzpsTfW+hXoMlghx33xN+fjew2N0fcvfPALOjD01kmKlEIBnuoZV1bKhv5mMXzWHMmJH7ex00EZhZbxvCRcBvE9al8kSySGbQcwQSA3V7W/ivn6+l9rgKLj1l0pF3GEaDJYIfAU+Z2f8BrcDvAcxsNkH10BGZ2SVmts7M1pvZJ5Osf4+ZPR++/mhm84/iZxA5AlUNSWbr6u7hn368Cnf46rsWYCNceh2s19AXzewJYDLwq4RhqMcAHz3Sgc0sB7gDuBioA5ab2SPuvjZhs03Aee6+18wuJWh7OPvofhSRAfTlASUCyUzffHIDyzfv5avvms/0yqIRP/+gVTzu/kySZS+neOyFwHp33whgZkuAK4G+RODuf0zY/hmgJsVjixwFJQLJPM++upevPfEKV8yfwlWnT01LDKmMNXS0phJMatOrLlw2kA8Bv0i2wswWmdkKM1tRX18/jCFKdlAbgWSmvc0d3LxkFZPKCvjCVaeMeJVQrygTQbKfKOn/SDO7gCAR/Guy9e6+2N1r3b22unrkulTJKKHnCCQD7Wvp4AP3LWdHYxtfv24BZQW5aYslyt4/dcC0hO81wPb+G5nZacA9wKXu3hBhPJK11FgsmWVLQzM3fGc5dXtbuf360znzuPQ+oxtlIlgOzDGzmcA24Frg+sQNzGw68FPgfUNoexA5OsoDkgFWbtnDh7+7kh53fvDhszlrRvoHaogsEbh7l5ndBDwO5AD3uvsaM7sxXH8n8B9AJfDNsG6sy91ro4pJspSeI5AM8ehz2/n4T55jSnkB37lhITOritMdEhDxg2HuvhRY2m/ZnQmf/w74uyhjEFHVkKRbZ3cP33jiFb7+2/XUHlfB4vfXMr44c6aA1xPCMvqpsVjSaP2uJj7+wCqeq9vPO8+o4YtXnUJBbk66wzqEEoFkAZUIZOS1d3Vz11MbuX3Zeorycrjj+jN462mT0x1WUkoEkj1UIpAR4O785sVdfPHna9nc0MJbT5vMZ992MhNKo59y8mgpEcjop8ZiGSGrtu7j1l+8yDMb9zCrupjvfnAhbxqByeePlRKBZAFVDUm0/vrqXm7/7XqeeGkXlcV5/OeV87hu4XRyc6J8Znf4KBHI6NfVHryPzdyiucRPd4/zmxd38p2nN/HMxj1UFOXyz285gQ+cO5OS/HhdWuMVrcjR6EsE+emNQ0aFxrZOHli+lfv/tJmte1qZOq6QT102l+vPPi52CaBXPKMWGYqu1uBdJQI5Bq/sPMD3n9nCgyvraO7o5qwZFXzq0pO4+OSJjI1JFdBAlAhk9OstEeQqEcjQ7G/p5NHnt/PQs3X89dV95OYYbzttCjecO5NTa8rTHd6wUSKQ0a+rLXhXiUBS0NzexRMv7eLnz29n2Uv1dHT3cMLEEj512VzecUYNVSWjr4pRiUBGP7URyBE0t3fx5Lp6fv7Cdn770i7aOnuYUJrPe143nXecXsMpU8vSNlfASFAikNGvU20Ecrite1pYtm4Xy17axdMbGujo6qGqJJ931U7jradOpnbGeHLGjN6LfyIlAhn91H1UgI6uHlZs2cOyl3axbF0963c1AXBcZRHvOXs6bzl5EgtnZs/FP5ESgYx+XW2Qk68hJrJMT4/z4o5G/rShgT+s381fNu2hpaObvJwxnH38eK5bOJ0LTqzm+OqSdIeadkoEMvp1tak0kAU6u3tYs72Rv2xq4C+b9rJiyx72tXQCMKu6mKvPrOENs6s4d3YVxTHt7x8V/TZk9OtqU9fRUWhvcwertu7j2Vf3snLLXv766j5aO7sBmFFZxMUnTeR1x1dyzuxKJpcXpjnazKZEIKNfV7t6DMVca0c3a1/bz3Nb9/N83T6eq9vPpt3NAOSMMeZOKuXdZ02jdkYFC2eMZ0KZEv9QKBHI6KeqoVjZ19LBuh0HWLO9kdXb97NmWyOv7DpATzh24MSyfE6rGcc1tTWcMb2C02rKKcrTpexY6Lcno19nm0oEGcbd2dHYxvpdTYe8NtQ3sbupo2+7qpJ8TplaxlvmTeTUqeXMnzaOibrbH3aRJgIzuwT4GsHk9fe4+6391s8FvgOcAXza3f8nyngkS3W1wVjVEadDV3cPr+5pCS709eHFflcTG+qbaWrv6tuurGAssyeUcOHcCcyeUMKciaXMm1KW0ZO5jCaRJQIzywHuAC4G6oDlZvaIu69N2GwP8DHg7VHFIaI2gui1dXazIeFC33vR37y7hY7unr7tJpblM3tCCe88YyqzJ5Qwa0IJsyeUUF2SP6qf3M10UZYIFgLr3X0jgJktAa4E+hKBu+8CdpnZWyOMQ7JdVxsUVaY7itjr6XF2HWhnS0MzmxuaD1bp1DdRt7e1byK4MQbTxxcxe0IJF8ydwOzqkr6LfllBbnp/CEkqykQwFdia8L0OOPtoDmRmi4BFANOnTz/2yCS7qPtoyprbu9i6t4Wte1p5dU8LW8PXq+Grvevg3X3e2DEcX1XM/JpxvPOMGmaHd/czKospyM1J408hQxVlIkhWzjuqyWPdfTGwGKC2tlYT0MrQqNdQn8a2TrbtbWXb3lbq9rawbV8r2/a1Urc3eO1p7jhk++K8HKaNL2JGVTHnnVDNcZVFTK8sZmZlMVMrCrNyOIbRKMpEUAdMS/heA2yP8HwiyWVBG0FPj7OnpYMd+9vY2djGa+H7jv1t7Eh4P9DWdch++WPHMLWikKnjCpk3pZxp4wuZVlHE9PFFTBtfREVRrurus0CUiWA5MMfMZgLbgGuB6yM8n0hyna2x7jXU0dXDzsbDL/CvNbaxM7zA72xso7P70MLyGIPq0nwmlRdyfHUx58yqZPK4QmrCC39NRRFVJXm60Et0icDdu8zsJuBxgu6j97r7GjO7MVx/p5lNAlYAZUCPmd0MnOzujVHFJVnGHTqaIK8o3ZEkdaCts+9u/bX9By/sOxIu8In96nsV5I5hUlkBk8oLqD2ugonlBUwOv08sK2ByeSFVJXmxn0JRRkakzxG4+1Jgab9ldyZ83kFQZSQSjY4m6O6AoqoRPW1Xdw8NzR1Jq2f6LvL722ju6D5s34qiXCaVFzKpLJ/TasrDC3twgZ9UXsDkskLKCsfqTl6GjZ4sltGteXfwPgzdR92dA+1d1B9oP/TV1M6uxuA9WNZGQ3NHX3fKXmPHGBNK85lUXsDcSaWcd0L1wQt8eBc/oSxfPW5kxCkRyOjWtCt4L5k44CbN4cV9d3gh732vb+o4bHli98leY8cYVSX5TCjLZ+q4AhZMK6e6tIDq0nwmhhf+SeUFVBXnM0a9bCQDKRHIqOPuNHd0s/tAO90b1jELeHSzs37zy9Q3tbO79+Le1M7uAx19QxcnMoPK4jyqSvKpKslnZlUxVSV5VJfmB6+S4EI/oTSfcepZIzGnRCCx4O7sb+0M7847aGjuvaB3sLup98Lewe4D7TQ0t9PWGdy535jzNJ/MhU8+sZ8We4WKojyqS/KpKs3jjOkV4ef8fu95jC9SQ6tkDyUCSZvuHmdPc+9FPfGCfvB777qG5vbDukdC0EVyfHF+39368eGde1VJPpUl+Zy75qd0vVbFslsuZ3yxLu4iySgRyLDq6Oo57MKeeNfekPB5T3NH3xjzifJyxlBVkkdlSXCHPndSGdWl+VQWBxf73uqaypI8KoryBn+6deUGmHiSJioRGYQSgRxR4sW9vqktfD/YgHqwMbWD/a2dSY9RlJcTXsDzmDa+iNOnV1BdkkdVeGGvLD74uaxgmLpGdnXAzrVw9qJjP5bIKKZEkKXcnab2LnY2trOrsY1dB9rZGb739ZoJu0b2TgDeX0n+2PAOPY8TJpZy7uyDd+tVvRf54qDOPS0zSG1/FrrbYdpRjXUokjWUCEah4AIfPJVaH17gdza2H7zYh9+T9ZYpzM3p6w0zq7qEs48fz4TSgkMu7tUlQc+ZjO/vvuG3YGPguHPTHYlIRlMiiJHmvgt8O7sOtLGr8eBdfO/7rsbkT6v2DkkwobSAU6aWc9FJBUwozWdiWQETysL30nxK8kfJE6vusPqnQWmgaHy6oxHJaEoEGaCl42AVzc7wYn7IBT68m0+c2q9XQe4YJpYVMLG0gJOnlHHBiROYWBY83DSxtIAJ4YW+dLRc4FO16XfQ8Aq84eZ0RyKS8ZQIIubu1De19433Xre3he37Wtm+ry18b6Wx7fALfP7Y8AJfls9Jk8s478SDd+29y6tLC4avYXU0cYenvhQ8TXzK1emORiTjKREMg67uHrbta2XT7mY27W5mS0PLwdmd9rb0PdzUa1xRbt8wwAtnjg9GjCwt6LvATyjTBf6YvPAgbHkaLvsfzUwmkgIlgiFqbOtk7fZGVm/bz9rtjazZ3sim3c2HTNDdO6vTzKpi3nRCNdMqCpk2voiaiiJqKgopztevPTLbV8HPb4GptVD7wXRHIxILuiINYk9zB6u37Wf19v2s2d7Imm372dzQ0rd+Ylk+86aUc/7camZVlTCzupgZlcWa7CNdNj4JS94LhePgmvtgTIb3ahLJEEoE/bza0MKjz2/nsedf48XXDs6PM218IadMKefqM2uYN7WceVPKmFCqaoeMsHcz/PYL8MJPoHouvPenUD413VGJxIYSAbBjfxuPPb+dR59/jee27gPgzOMq+MTfnMjp08Yxb0o55UW5aY5SDrN3M/z5LvjL3cHd/xtugTfeAvml6Y5MJFayNhF0dvfw8F+38dDKOv6yeQ/ucMrUMv7t0rm89bTJ1FRk5tSGWa2zNWgD2PgkrP81bFsZPDC24D1wwaegbEq6IxSJpaxMBE+v382//2w1m3Y3M6u6mJsvOoHL509mVnVJukMTgI5mOLAjuGmnm9cAAAxzSURBVOOvXwc718DOF4L3ni7AYOqZcNFn4dRrYNy0dEcsEmtZlwju+f1Gvrj0RWZUFnPP+2u56KQJatiNWk93cDff2QItDcGrdW/watoFzfXQuB32b4V9rwbrExVXw4ST4ZyPQs1CmP46PS0sMowiTQRmdgnwNSAHuMfdb+233sL1lwEtwAfc/dmo4vnJiq184ecvctmpk/jKNQsozFOvEnp6oKcTutqgoyW4WHe1QVd7MOl7V3vwam8M1nW2Beu7O6C7MxjUrbsT2g9A2/7gvaMJ2hqhdU+wrKtt8Bjyy4KHv8ZNh8nzg/fSycF71QlQMmFkfhciWSqyRGBmOcAdwMVAHbDczB5x97UJm10KzAlfZwPfCt+HXXtXN/+79DkuPC6Pr18xnbGde6GjJ3gK1XuA8D3pdz/0e08n7N8WXAQP2Yfwe8KyZMfu3ba7I7gT7uk6uD5pHIkxJIkv8XNXG7Q3hRfq3ot1/8/he09nWNVyDHLyglduIRRWBA21eSVQWQ2FtUFXzrwSyC06uE1xVfBeWBHc7ecWHlsMInJMoiwRLATWu/tGADNbAlwJJCaCK4HvursDz5jZODOb7O6vDXcwm373I57s/hh5O7vhK8N99GOQkwdjC4NJcs2Cxk/6fx4TfO/7zMDbjckJLrB5xZBTATm5By/WY8P3Mbnh8tyDn8cWQF5RcMEemx98z8k7+Dm/LLhg5xYGy3Lyg/1UrSYSe1EmgqnA1oTvdRx+t59sm6nAIYnAzBYBiwCmT59+VMG0TjqLFSXnc8brLqQgLze8eHLwItv/opv0e8L2BeVBdUb/bZId95Bj9Tt+QTnkZF1TjYhkkCivQMluFftPTJjKNrj7YmAxQG1tbZLJDY/s9JNPhJN/ejS7ioiMalHO5F0HJPbrqwG2H8U2IiISoSgTwXJgjpnNNLM84FrgkX7bPAK83wKvA/ZH0T4gIiIDi6xqyN27zOwm4HGC7qP3uvsaM7sxXH8nsJSg6+h6gu6jN0QVj4iIJBdpK6W7LyW42CcuuzPhswMfiTIGEREZXJRVQyIiEgNKBCIiWU6JQEQkyykRiIhkOQvaa+PDzOqBLUe5exWwexjDiUIcYoR4xKkYh4diHB7pjvE4d69OtiJ2ieBYmNkKd69NdxyDiUOMEI84FePwUIzDI5NjVNWQiEiWUyIQEcly2ZYIFqc7gBTEIUaIR5yKcXgoxuGRsTFmVRuBiIgcLttKBCIi0o8SgYhIlsuaRGBml5jZOjNbb2afTHc8/ZnZvWa2y8xWpzuWgZjZNDNbZmYvmtkaM/vHdMfUn5kVmNlfzOy5MMbPpzumgZhZjpn91cweS3csyZjZZjN7wcxWmdmKdMeTTDi97YNm9lL4d/n6dMfUn5mdGP4Oe1+NZnZzuuNKlBVtBGaWA7wMXEwwGc5y4Dp3XzvojiPIzN4ENBHM4XxKuuNJxswmA5Pd/VkzKwVWAm/PsN+jAcXu3mRmucAfgH9092fSHNphzOwWoBYoc/fL0x1Pf2a2Gah194x9UMvM7gd+7+73hPOeFLn7vnTHNZDwWrQNONvdj/bB2GGXLSWChcB6d9/o7h3AEuDKNMd0CHf/HbAn3XEMxt1fc/dnw88HgBcJ5pjOGB5oCr/mhq+Mu9sxsxrgrcA96Y4lrsysDHgT8G0Ad+/I5CQQugjYkElJALInEUwFtiZ8ryPDLmBxY2YzgNOBP6c3ksOFVS6rgF3Ar90942IEbgP+BehJdyCDcOBXZrbSzBalO5gkjgfqge+EVWz3mFlxuoM6gmuBH6U7iP6yJRFYkmUZd5cYF2ZWAjwE3OzujemOpz9373b3BQRzYC80s4yqajOzy4Fd7r4y3bEcwbnufgZwKfCRsPoyk4wFzgC+5e6nA81AxrX/9Qqrrq4AfpLuWPrLlkRQB0xL+F4DbE9TLLEW1rs/BPzA3X+a7ngGE1YTPAlckuZQ+jsXuCKsg18CXGhm309vSIdz9+3h+y7gYYIq1kxSB9QllPgeJEgMmepS4Fl335nuQPrLlkSwHJhjZjPDrHwt8EiaY4qdsCH228CL7v7VdMeTjJlVm9m48HMh8GbgpfRGdSh3/zd3r3H3GQR/i7919/emOaxDmFlx2CGAsLrlLUBG9Whz9x3AVjM7MVx0EZAxHReSuI4MrBaCiOcszhTu3mVmNwGPAznAve6+Js1hHcLMfgScD1SZWR3wWXf/dnqjOsy5wPuAF8I6eIBPhXNTZ4rJwP1h74wxwAPunpHdMzPcRODhIPczFvihu/8yvSEl9VHgB+EN3kbghjTHk5SZFRH0Wvz7dMeSTFZ0HxURkYFlS9WQiIgMQIlARCTLKRGIiGQ5JQIRkSynRCAiksGGOiClmb3LzNaGgy7+MJV9lAhkRJnZp8M/0OfDkRjPHuL+HzCzKUPcZ8ZQRnU1s6W9zyKkuP0VvSPamtnnzOyfhxhf4v5vN7OTh7j/f5rZm4e4z2YzqxrKPpI295HiQ5FmNgf4N4KnwucBKY1ymhXPEUhmCIcIvhw4w93bwwtR3hD2zwE+QPBgU2RPhrv7ZUPc/hGO8gFFMxvbb/+3A48xhAej3P0/jubcEg/u/rtwbK8+ZjYLuAOoBlqAD7v7S8CHgTvcfW+4765UzqESgYykycBud28HcPfdvcMYmNlF4cBhL4RF4fxw+WYz+w8z+wPBk5m1BA8QrTKzQjM708yeCgdGezwcKptw+XNm9ifgI8mCMbPJZva78FirzeyNCeesCksSL4WDma02sx+Y2ZvN7Gkze8XMFobbf8DMbk9y/A+b2fIwjofCh4ows/vM7Ktmtgz4Uu/+ZnYOwVg0Xw5jmmVmzyYcb46ZHTY+UXi8qxNi/7yZPRv+LueGyyvN7Ffh7/guEsbfMrP3WjCHwyozu8uCQfvOCkttBeFTxmssw8ZsynKLgY+6+5nAPwPfDJefAJwQ/o0+Y2YplSSUCGQk/QqYZmYvm9k3zew8CCaTISj+vtvdTyUoqf5Dwn5t7v4Gd/8+sAJ4TzioXBfwDeDq8D/EvcAXw32+A3zM3QebqOR64PHwWPOBVUm2mQ18DTgNmBvu8waC/3yfOsLP+1N3P8vd5xMM2f2hhHUnAG9294/3LnD3PxKUDD7h7gvcfQOw38wWhJvcQPB7OpLd4WBx3wrjBPgs8IdwcLZHgOkAZnYS8G6CqoQFQDfB73d5uN0XgP8HfN/dM2qIiWxlwaCP5wA/seAJ/7sIbrIg+L8zh2CUguuAe1Kp5lTVkIyYcLKYM4E3AhcAPw7rxv8KbHL3l8NN7ye4i78t/P7jAQ55InAK8GsLhkLIAV4zs3JgnLs/FW73PYIBv/pbDtxrwUB6P3P3ZIlgk7u/AGBma4An3N3N7AVgxhF+5FPM7AvAOKCEYIiTXj9x9+4j7A/BfAU3WDCJzbtJbeC33sEAVwLvCD+/qfezu//czPaGyy8CzgSWh7/DQoLhuwH+k+B31AZ8LIXzysgYA+wLE3d/dcAz7t4JbDKzdQSJYfmRDigyYsIhop90988CNwHvJPkw4YmaB1huwJrw7nmBu5/q7m8Jlx9x7JRwMqA3EcwY9T0ze3+SzdoTPvckfO/hyDdS9wE3haWczwMFCesG+pn6e4ggiV0OrHT3hhT26Y2xu1+MyX4nBtyf8Ds80d0/F64bT5DASvvFLmkUDv2+ycyugWAwSDObH67+GcFNFmEb3AkEYzANSolARowFc7fOSVi0ANhCMDroDDObHS5/H/BU//1DBwguTADrgOqwERozyzWzeeHw0/vN7A3hdu8ZIJ7jCOYFuJtgVNXhHsK4lKCEkjtQDEkk/ny4extBSeJbBNVdR+t3vTGY2aVARbj8CeBqM5sQrhsf/l4gqIf+DPAD4EvHcG45BhYMSPkn4EQzqzOzDxH8W37IzJ4D1nBwxsXHgQYzWwssI6hmPOLNg6qGZCSVAN8I6yy7gPXAIndvM7MbCOo8xxIUY+8c4Bj3AXeaWSvweuBq4OthddBYguqkNQT16feaWQuHVskkOh/4hJl1EswXnaxEcCw+QzCD2xbgBRIu8INYAtxtZh8jaPvYQHAhfgdBG8vR+jzwo7Dx+SngVQB3X2tm/04wE9kYoJNgEprzgC53/6EFvbX+aGYXuvtvjyEGOQruft0Aqw5rCPZgFNFbwlfKNPqoSIaz4LmEcnf/TLpjkdFJJQKRDGZmDwOzgAvTHYuMXioRiIhkOTUWi4hkOSUCEZEsp0QgIpLllAhERLKcEoGISJb7/5VnEJCAGRBgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as pyplot\n",
    "\n",
    "\n",
    "pyplot.plot(W_sparse_CF_sorted, label = 'CF')\n",
    "pyplot.plot(W_sparse_CBF_sorted, label = 'CBF')\n",
    "pyplot.ylabel('Similarity value')\n",
    "pyplot.xlabel('Sorted similarity index')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's now plot the delta for the common values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get common structure\n",
    "W_sparse_CF_structure = W_sparse_CF.copy()\n",
    "W_sparse_CF_structure.data = np.ones_like(W_sparse_CF_structure.data)\n",
    "\n",
    "W_sparse_CBF_structure = W_sparse_CBF.copy()\n",
    "W_sparse_CBF_structure.data = np.ones_like(W_sparse_CBF_structure.data)\n",
    "\n",
    "W_sparse_common = W_sparse_CF_structure.multiply(W_sparse_CBF_structure)\n",
    "\n",
    "# Get values of both in common structure of CF\n",
    "W_sparse_delta = W_sparse_CBF.copy().multiply(W_sparse_common)\n",
    "W_sparse_delta -= W_sparse_CF.copy().multiply(W_sparse_common)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_delta_sorted = np.sort(W_sparse_delta.data.copy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEGCAYAAACgt3iRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8den2fe0SdqmTWlaqIUWSukCIosK6ACyKTjiKAKDdPghOo4zoziMCC6PhyLjMA4qlEVQUBRwQYaRHUQYgRYoS6G0dKGhS5Yu2df7+f1xTtrbNElvcpOcm+T9fDzuI+eee849n1ya++b7/Z7zPebuiIiIJGNC1AWIiMjopzAREZGkKUxERCRpChMREUmawkRERJKWHnUBw6G0tNQrKyujLkNEZNRYuXJlrbuXDXb/MRkmlZWVrFixIuoyRERGDTPblMz+6uYSEZGkKUxERCRpChMREUnamBwz6U1HRwdVVVW0trZGXcqIyM7OpqKigoyMjKhLEZFxYNyESVVVFQUFBVRWVmJmUZczrNyduro6qqqqmDVrVtTliMg4EGk3l5mdamZrzGydmV3Zz3ZLzazLzM4b7LFaW1spKSkZ80ECYGaUlJSMm1aYiEQvsjAxszTgx8BpwDzg02Y2r4/tvg88PATHTPYtRo3x9LuKSPSibJkcDaxz9/Xu3g7cA5zdy3ZfBO4HqkeyOBGR0eTR1du56el3Ijt+lGEyHdgc97wqXLeHmU0HPg7cdKA3M7NlZrbCzFbU1NQMaaHD4ZprruH6669P6PU77riDLVu2jFRpIjIKPfFWNbf9ZUNkx48yTHrrh+l5p64bgK+5e9eB3szdl7v7EndfUlY26BkBUpLCRERSXZRhUgXMiHteAfT8xlwC3GNmG4HzgJ+Y2TkjU97Q++53v8vcuXM55ZRTWLNmDQDvvPMOp556KosXL+aEE07grbfe2mef++67jxUrVvCZz3yGhQsX0tLSwre+9S2WLl3K4YcfzrJly9DdMkUkalGeGvwiMMfMZgHvAecDfxe/gbvvOa/VzO4AHnT33yd74Gv/+Aart9Qn+zb7mDetkG+eOb/P11euXMk999zDyy+/TGdnJ4sWLWLx4sUsW7aMm266iTlz5vD8889z+eWX88QTT+zZ77zzzuPGG2/k+uuvZ8mSJQBcccUVXH311QBccMEFPPjgg5x55plD+vuIiAxEZGHi7p1mdgXBWVppwO3u/oaZXRa+fsBxktHkmWee4eMf/zi5ubkAnHXWWbS2tvLcc8/xyU9+cs92bW1tB3yvJ598kuuuu47m5mZ27NjB/PnzFSYiEqlIL1p094eAh3qs6zVE3P2ioTpufy2I4dTzdN1YLEZxcTGvvPJKwu/R2trK5ZdfzooVK5gxYwbXXHONricRkchpbq4RcuKJJ/K73/2OlpYWGhoa+OMf/0hubi6zZs3i3nvvBYIr11etWrXfvgUFBTQ0NADsCY7S0lIaGxu57777Ru6XEBHpg8JkhCxatIhPfepTLFy4kHPPPZcTTjgBgLvvvpvbbruNI488kvnz5/OHP/xhv30vuugiLrvsMhYuXEhWVhaXXnopRxxxBOeccw5Lly4d6V9FRGQ/NhbPBFqyZIn3vDnWm2++yWGHHRZRRdEYj7+zyHj19d++xmNvbufFq04Z1P5mttLdlwz2+GqZiIhI0hQmIiKStHEVJmOxS68v4+l3FZHojZswyc7Opq6ublx8yXbfzyQ7OzvqUkRknBg3N8eqqKigqqqK0TAJ5FDovtOiiMhIGDdhkpGRobsOiogMk3HTzSUiIsNHYSIiIklTmIiISNIUJiIikjSFiYiIJE1hIiIiSVOYiIiMCdFekK0wEREZA9xhgh14u+GiMBERGQNi7kyw6NJEYSIiMgbEHIWJiIgkJ+ZOhFmiMBERGQvcUZiIiEhyXGMmIiKSLI2ZiIhI0jRmIiIiSXO1TEREJFkxdyJsmChMRETGArVMREQkaRozERGRpOlsLhERSZq7MyHCb3SFiYjIGBAMwKtlIiIiSYhpCnoREUmWAzZex0zM7FQzW2Nm68zsyl5e/4yZvRo+njOzI6OoU0Qk1QVzc0V3/MjCxMzSgB8DpwHzgE+b2bwem20APujuC4BvA8tHtkoRkdFhPN8c62hgnbuvd/d24B7g7PgN3P05d98ZPv0rUDHCNYqIjAqx2Pg9NXg6sDnueVW4ri+XAP/b14tmtszMVpjZipqamiEqUURkdIi5E+V8KlGGSW+/tve6odmHCcLka329mbsvd/cl7r6krKxsiEoUERkdnGjP5kqP7tBUATPinlcAW3puZGYLgFuB09y9boRqExEZVYKLFqNrH0TZMnkRmGNms8wsEzgfeCB+AzM7CPgtcIG7vx1BjSIio0LU06lE1jJx904zuwJ4GEgDbnf3N8zssvD1m4CrgRLgJ+H5053uviSqmkVEUlXUEz1G2c2Fuz8EPNRj3U1xy58HPj/SdYmIjDYxH8cXLYqIyBAZrxctiojI0Il6zERhIiIyBsTUMhERkWRpzERERJLm7lFeAK8wEREZC1xjJiIikqyYbtsrIiLJCi5aVMtERESSoG4uERFJWkwD8CIikqzgosXojq8wEREZA5zxe9teEREZIrGYLloUEZEkdXTFyExP8TAxs5lmdkq4nGNmBcNbloiIDERHV4z0VL7TopldCtwH3ByuqgB+P5xFiYjIwHR2ORlpKRwmwBeA44B6AHdfC0wezqJERGRg2rtiZKSldjdXm7u3dz8xs3TAh68kEREZqM5Y6rdMnjazfwNyzOwjwL3AH4e3LBERSVQs5nTFnPQUb5lcCdQArwH/QHDP9n8fzqJERCRx7V0xgEhbJukH2sDdY8At4UNERFJMW2cQJlnpKRwmZraBXsZI3H32sFQkIiID0tbRBUB2RlpkNRwwTIAlccvZwCeBScNTjoiIDFRLCoTJAdtE7l4X93jP3W8AThqB2kREJAGtHUE3V3ZGandzLYp7OoGgpaIr4EVEUkRrd8skPbW7uf4jbrkT2Aj87bBUIyIiA9aaAt1ciZzN9eGRKERERAanuT0Ik7ysFAwTM/tKfzu6+w+HvhwRERmoxrZOAPKyEulsGh79HVnjIiIio0BTKoeJu187koWIiMjgNLQGYVKQnYJh0s3MsoFLgPkE15kA4O5/P4x1iYhIgnY2t5M+wSiIsGWSyEnJvwCmAn8DPE1wP5OG4SxKREQSt7O5g+LcjJS/be8h7v4NoMnd7wQ+BhwxvGWJiEiidjS1MTE3M9IaEgmTjvDnLjM7HCgCKoetIhERGZCahjbKCrIirSGRMFluZhOBbwAPAKuB7w/Fwc3sVDNbY2brzOzKXl43M/tR+PqrPa7GFxERYHt9G5MjDpNERmt+5u5dBOMlQzZTsJmlAT8GPgJUAS+a2QPuvjpus9OAOeHjGOCn4U8REQHaO2Nsq29l+sScSOtIpGWywcyWm9nJNrSjO0cD69x9fXhb4HuAs3tsczbwcw/8FSg2s/IhrEFEZFR7ZfMuumLOEdOLI60jkTCZCzwGfAHYaGY3mtnxQ3Ds6cDmuOdV4bqBbgOAmS0zsxVmtqKmpmYIyhMRSX1/fruGCQbHHlwSaR2JTEHf4u6/cfdPAAuBQoIur2T11srpeROuRLYJVrovd/cl7r6krKws6eJERFJdY1snv3zhXY6fU0ZRTkaktSQ0+b2ZfdDMfgK8RHDh4lDMGlwFzIh7XgFsGcQ2IiLj0q3PrGdHUztf+cj7oi7lwGES3rb3y8AzwOHu/rfufv8QHPtFYI6ZzTKzTOB8grPF4j0AfC48q+v9wG533zoExxYRGdX+9PpWfvT4Wj62oJyFM6IdL4HEzuY60t3rh/rA7t5pZlcADwNpwO3u/oaZXRa+fhPwEHA6sA5oBi4e6jpEREaTusY2bv7zem55Zj1HzSjmunMXRF0SkNj9TIY8SOLe+yGCwIhfd1PcshMM/IuIjGtrtjXw8//byH0rq2jrjHH+0hl888z55GRGdw+TeNHNCiYiIn3q6IrxatUunnirmoff2M666kYy0ydw7qLpXHL8bA6ZnB91iftIZNbgtPCiRRERGSbV9a2sqtrNqs27WFW1i5c27aSpvYu0CcbRlZP43LEzOf2Ickrzo73SvS+JtEzWmdl9BFfCrz7g1iIi0id3p7qhjbe2NbB6S/2e8Ni6uxWAtAnGoVML+MSiCo49uITjDi6lKDfa034TkUiYLCA40+pWM5sA3A7cM5xjKSIio11bZxebdzSzobaZ9TWNbKhtYl11I29vb6A+vJkVQGVJLksrJ3HkjGIWzihiXnlRyoyDDEQiA/ANwC3ALWZ2IvAr4D/D1sq33X3dMNcoIpJy2jtjbK9vZVt9K1t3t7JlVwub6prZVNfEprpmtuxuweMusS7Jy+TgyfmceeQ05kzOZ+7UQg4rL6A44qnjh0pCYyYE9zC5mGDq+f8A7gZOIDgTK/qrZUREhlBbZxfbd7exdXcLW3cHYbEtXO4Oj9rGtn3CAmBSXiYzS3JZWjmRmSUVVJbmUlmSx+zS/FHRVZWMRLq51gJPAj9w9+fi1t8XtlREREaNlvauMBBa2BYGRfzytt2t1DW177dfQXY604pymFqUzbzyQqYWZVNelM3UohzKw+WC7LEdGP1JJEw+5+5/iV9hZse5+7Pu/qVhqktEZEC6Ys6OpnZqGtqoaWyjuj4Ihq3dP8PQ2NXcsd++xbkZTC0MAuHIGcWUF2aHYRGEx9SibPIjvL/6aJDIp/MjoOdNqf67l3UiIkMqFnN2NrdT29hObWNb+AiXG9qobmjbEx51jW3EepkGtiQvk6lF2UwvzmbxzGLKw5bEnrAozB6VA96pps8wMbNjgQ8AZWb2lbiXCgmmPxERGbDWji5qG9uoC0OhrrGd2qbgZ11cWNQ1tbOjqZ2uXhIifYJRmp9FaUEQFAsqiigryAoe+VmUFmQxuSCLKYXZZGfo62ok9NcyyQTyw20K4tbXA+cNZ1EiMnp0dsXY0dwehkE7dWEw7GgKlmvDkKhraqe2oY2m9t6vgc7LTKMkP4uS/EwqJuZy1EHFlOQFz0vzsyjNz6KsIFguyslgaO/VJ8nqM0zc/WngaTO7w903jWBNIhKhWMzZ1dKxJwC6A6K2sZ0de1oQYWg0tfc6BgEwwYKzm0rDgFg4KT4cutdnURJuo66m0a2/bq4b3P3LwI1mtl87093PGtbKRGRIuDuNbZ1hS6E9rnspbDU0BS2HHU3tewKjt7EHMyjOydgTAHOnFuwJh5K8zD3rS/IzmZSXRXFOBhMmqPUwXvTXzfWL8Of1I1GIiCSurbMrCIe4cYfulkJdj26l2qZ22jtjvb5PQXY6JXmZTMrL5KBJ+3YtxYdDSV4WE3MzSE9L6H56Mg711821Mrxg8VJ3/+wI1iQy7rg79a2d+wxAx5+5VLcnMILnDXHTccTLTJsQBkHQdXTI5HzK8rOYFNdy6O52KsnPJCtdXUsyNPo9Ndjdu8yszMwy3X3/q3hEpE/d3Us1DUEo1DS0UdPQunc5DIyahiAo2rv2bz2YwaTcva2D+dMKKQ3DYU8ohMuT8jMpyErXwLREIpHrTDYCz5rZA0BT90p3/+FwFSWSyrpiTl1jcI1DdUMrNQ1tVNfvveahuqGVmjAkWjv2D4i0CbYnAEoLspgzuYDSgkzK8vc9c6k7NNI07iCjQCJhsiV8TGDfU4RFxpSOrhg1DW1sr29le33QitheH4RDdVxg9DVAXZSTQVl4fcPigyZSVhAEwuTCveEwuSCLibmZGpiWMSeRWYOvHYlCRIZLLObUNbWHIREExPb67oAIJu7bXt9GXdP+E/dNMMKA6J5qo4iy/CzKCrMpC4NichgaujhOxrNEZg0uA74KzAeyu9e7+0nDWJdIQrpiTnXD3gn69pndNXxe3dBKR9e+KWEWTLNRVpDN1MIsFlQUMbkgmymF2UwpDMJjSlEWJXlZ6mYSSUAi3Vx3A78GzgAuAy4EaoazKBEIBrB3NXfw3q6WPfeL2LK7hS27Wtm6q4Utu1rY3tC233QbWekTKC8KgmFp5cQ9s7pOKcwKwyKbsoIsMnSaq8iQSSRMStz9NjP7x7ir4p8e7sJk7OvsirGtvpWqnS1U7WzhvZ0tvLermS27WsPQaNlvADszbQLlxdlMK8rh/QeXMK0oh/LicCrwwiA0inM11YbISEskTLrnSthqZh8jGIyvGL6SZKyIxYJ7Xb+7o5l3dzSzeUczm3c27wmObfWt+7UqygqymF6cw2FTCzlp7mSmFecwrTibacU5lBflUJKnwWuRVJRImHzHzIqAfyaYer4Q+KdhrUpGjY6uGFU7W9hY18TG2qY9ty19d0czm3e27HPltRlMLcymYmIOR8+axPTiHKZPzKFiYg4VE3MpL9IMryKjVSJncz0YLu4GPjy85Ugqcne217exvqaRd2qb2FDTxPraRtbXNPHerpZ9Whd5mWkcVJLHnMkFnHzYFGZMyuWg8DGtOFtXXIuMUf1N9PjfQC9n0wd0l8Wxp7Wji3XVjbxTEwTF+tom1tc0sqG2iea4acNzM9OYVZrHERVFnL1wGgdNymVWaR6VpXmU5GVqvEJkHOqvZbJixKqQEdXZFWNjXRNvbm1gzbYG1mxvYO32BjbtaN5znYUZVEzMYXZpPkfPmsTssnxml+YxuyyPqYXZCgwR2Ud/Ez3eOZKFyPBoaO3gjS31vLGlnre21vPmtnre3t64ZywjbYIxqzSPedMKOXvhdN43pYBDJuczsyRX4xcikrAD3s/EzP5IL91dup9J6mnvjPHWtnpWbd7FK5t382rVLtbVNO5pbZTmZ3JYeSEXHjuTQ6cWcmh5AQeX5Ss0RCRpup/JKBWLOetrm1i1eRevVu3ilardvLmlfs/Ms6X5mRxZUcwZC6axYEYR86cVMrkg+wDvKiIyOP3ezyT8qQsUU0BnV4zXt9Tzf+/U8df1dby0aScNbcE9LfIy0ziiooiLj6vkyBnFHDmjmGlFGtcQkZGTyNxcZwDfBmaG2xvg7l44zLWNa+7O2upGnllby7Pranlhww4aw/CYMzmfsxZOY2EYHAeX5Wv+KBGJVCIXLd4AfAJ4zb3nnKoylBrbOnl2XS1Pranm6TU1bNndCsCs0jzOWjiNY2eX8P7ZJZQVZEVcqYjIvhIJk83A60MZJGY2iWDyyEqCm2/9rbvv7LHNDODnwFQgBix39/8aqhpSxa7mdh5ZvZ0/vb6Nv6ytpb0rRn5WOscdUsIXT57DCXNKqZiYG3WZIiL9SiRMvgo8FE7u2Na9Msk7LV4JPO7u3zOzK8PnX+uxTSfwz+7+kpkVACvN7FF3X53EcVNCc3snj67ezu9ffo9n1tbSGXOmF+fwuWNncvJhU1g8cyKZ6ZrRVkRGj0TC5LtAI8G9TDKH6LhnAx8Kl+8EnqJHmLj7VmBruNxgZm8C04FRGSbuzsubd3HPC+/yP69upam9i2lF2Vxy/Cw+tqCcI6YXacBcREatRMJkkrt/dIiPOyUMC9x9q5lN7m9jM6sEjgKe72ebZcAygIMOOmjICk1WW2cXD7yyhZ89u5HVW+vJy0zj9CPKOXdxBUdXTtIMuCIyJiQSJo+Z2Ufd/ZGBvLGZPUYw3tHTVQN8n3zgfuDL7l7f13buvhxYDrBkyZLITxRoaO3gzuc2csdzm6htbGPulAK+c87hnHPUdPKzEvnYRURGj0S+1b4AfNXM2gjubZLQqcHufkpfr5nZdjMrD1sl5UB1H9tlEATJ3e7+2wRqjVxHV4y7/rqJ/3p8LbuaO/jg+8q49ITZHHdIibqxRGTMSmQK+oJhOO4DBLf//V748w89N7Dgm/c24M0kB/tHhLvz5JpqvvM/b7K+ponjDinha6ceyoKK4qhLExEZdv3NzXWou79lZot6e93dX0riuN8DfmNmlwDvAp8MjzkNuNXdTweOAy4AXjOzV8L9/s3dH0riuMPi7e0NfPvB1TyztpbZpXncduESTjp0sloiIjJu9Ncy+QrBgPZ/9PKaAycN9qDuXgec3Mv6LcDp4fJfCLrUUlYs5tz+7Aa+/6e3yMlI4xtnzOOC98/Uab0iMu70NzfXsvCn7q7Yi5qGNv7l3lU8/XYNH503he+du4BJeUN15rSIyOjSXzfXUmCzu28Ln38OOBfYBFzj7jtGpsTUs3LTTi67ayW7Wzr49jmH89ljDlKXloiMa/31x9wMtAOY2YkE4xw/J7gX/PLhLy01PbZ6O5++5a/kZqbxwBXHccH7ZypIRGTc62/MJC2u9fEpgrmx7gfujxsQH1f+9Po2rvjlS8yfVsgdFx/NRHVriYgA/bdM0sysO2xOBp6Ie23cXXX34sYdfOlXL3NERRF3ff4YBYmISJz+QuFXwNNmVgu0AM8AmNkhBF1d48bW3S38wy9WUjExh59dtJSC7IyoSxIRSSn9nc31XTN7HCgHHombgn4C8MWRKC4VxGLOl+95hdaOLm657FiKc9UiERHpqd/uKnf/ay/r3h6+clLP7c9u4PkNO7juvAUcXJYfdTkiIilJV9f1o7q+lR8++jYnHTqZTy6uiLocEZGUpTDpx38+9jYdXTG+eeY8nf4rItIPhUkf3tvVwr0rqvi7ow9iZkle1OWIiKQ0hUkffvn8JmLuXHri7KhLERFJeQqTXnR2xfj1i1WcdOhkKibmRl2OiEjKU5j04oUNO6htbOPcRRp0FxFJhMKkF4++uZ2s9Al8aG6/t6YXEZGQwqQXT6+p4diDS8jJTIu6FBGRUUFh0kPVzmbW1zZxwpyyqEsRERk1FCY9rNy0E4BjZk2KuBIRkdFDYdLD6q31ZKZNYO7UgqhLEREZNRQmPbxT3cis0jwy0vTRiIgkSt+YPayrbuTgybriXURkIBQmcbpiTtXOFio1fYqIyIAoTOLUNLTRGXOmFedEXYqIyKiiMIlT3dAKwJTC7IgrEREZXRQmcXY2dwAwKU+35RURGQiFSZxdze0AujWviMgAKUzi7GgKwmSiwkREZEAUJnF2hd1cRTnq5hIRGQiFSZzWzi6y0ieQNkG36BURGQiFSZy2jhiZ6fpIREQGSt+ccdo6Y2Sla9p5EZGBUpjEaQu7uUREZGD0zRmnvTNGVoY+EhGRgYrkm9PMJpnZo2a2Nvw5sZ9t08zsZTN7cLjrUjeXiMjgRPW/4VcCj7v7HODx8Hlf/hF4cySKauvUALyIyGBE9c15NnBnuHwncE5vG5lZBfAx4NaRKKqtQ2MmIiKDEdU35xR33woQ/pzcx3Y3AF8FYgd6QzNbZmYrzGxFTU3NoIoKurkUJiIiA5U+XG9sZo8BU3t56aoE9z8DqHb3lWb2oQNt7+7LgeUAS5Ys8QGUuke7xkxERAZl2MLE3U/p6zUz225m5e6+1czKgepeNjsOOMvMTgeygUIzu8vdPztMJevUYBGRQYrqm/MB4MJw+ULgDz03cPevu3uFu1cC5wNPDGeQAMQcJmgqFRGRAYsqTL4HfMTM1gIfCZ9jZtPM7KGIasLdUZSIiAzcsHVz9cfd64CTe1m/BTi9l/VPAU8Ne12AKU1ERAZMAwRx3FHLRERkEBQmcRzH1DQRERkwhUkctUxERAZHYRLHHbVMREQGQWESx901AC8iMggKkziOurlERAZDYRIn6OaKugoRkdFHYRLHcUxtExGRAVOYxFHLRERkcBQmcXQFvIjI4ChM4riDhuBFRAZOYbIPR5MGi4gMnMIkTkxjJiIig6IwiRNMQa80EREZKIVJHA3Ai4gMjsIkjiZ6FBEZHIVJnGBuLsWJiMhAKUzieNQFiIiMUgqTeDqbS0RkUBQmcYJZg5UmIiIDpTCJ466LFkVEBkNhEkcXLYqIDI7CJI6js7lERAZDYRLn1PlTOay8IOoyRERGnfSoC0glN5x/VNQliIiMSmqZiIhI0hQmIiKSNIWJiIgkTWEiIiJJU5iIiEjSFCYiIpI0hYmIiCRNYSIiIkkz97F3Fw8zqwE2DXL3UqB2CMsZKqlaF6i2wUjVukC1DVaq1pZoXTPdvWywBxmTYZIMM1vh7kuirqOnVK0LVNtgpGpdoNoGK1VrG6m61M0lIiJJU5iIiEjSFCb7Wx51AX1I1bpAtQ1GqtYFqm2wUrW2EalLYyYiIpI0tUxERCRpChMREUmeu+sRdPWdCqwB1gFXDuH73g5UA6/HrZsEPAqsDX9OjHvt62ENa4C/iVu/GHgtfO1H7O2izAJ+Ha5/HqiM2+fC8BhrgQt7qW0G8CTwJvAG8I+pUB+QDbwArArrujYV6urx2aUBLwMPplJtwMbwPV8BVqRKbUAxcB/wFsG/t2NTpK654WfV/agHvpwKtYWv/xPB38DrwK8I/jZSorb9ah2qL83R/CD4YngHmA1kEnyJzRui9z4RWMS+YXIdYWABVwLfD5fnhcfOAmaFNaWFr70Q/gEa8L/AaeH6y4GbwuXzgV+Hy5OA9eHPieHyxB61lQOLwuUC4O2whkjrC98jP1zOCP+Rvz/qunp8dl8BfsneMEmJ2gjCpLTHushrA+4EPh8uZxKES+R19fI9sA2YmQq1AdOBDUBO+Pw3wEWpUFuvn99IfWGn8iP8kB+Oe/514OtD+P6V7Bsma4DycLkcWNPbcYGHw9rKgbfi1n8auDl+m3A5neBKV4vfJnztZuDTB6jzD8BHUqk+IBd4CTgmVeoCKoDHgZPYGyapUttG9g+TSGsDCgm+FC2V6url39pHgWdTpTaCMNlM8IWeDjwY1hh5bb09NGYS6P6P1q0qXDdcprj7VoDw5+QD1DE9XO6tvj37uHsnsBso6ee9emVmlcBRBK2AyOszszQze4Wgi/BRd0+JukI3AF8FYnHrUqU2Bx4xs5VmtixFapsN1AA/M7OXzexWM8tLgbp6Op+gK4lUqM3d3wOuB94FtgK73f2RVKitNwqTgPWyzke8ir7r6K++weyz70HN8oH7gS+7e30q1OfuXe6+kKAVcLSZHZ4KdZnZGUC1u6/sp55Iagsd5+6LgNOAL5jZiSlQWzpBV+9P3f0ooImgeybquvYe0CwTOAu4t5+6RrQ2M5sInE3QZTUNyDOzz6ZCbb1RmASqCHJm1eIAAAaWSURBVAaju1UAW4bxeNvNrBwg/Fl9gDqqwuXe6tuzj5mlA0XAjn7eax9mlkEQJHe7+29TrT533wU8RXCCRCrUdRxwlpltBO4BTjKzu1KkNtx9S/izGvgdcHQK1FYFVIWtSwgG4helQF3xTgNecvft4fNUqO0UYIO717h7B/Bb4AMpUtv++usDGy8Pgv9zWk/wfwDdA/Dzh/D9K9l3zOQH7DuAdl24PJ99B9DWs3cA7UWCQejuAbTTw/VfYN8BtN+Ey5MI+qknho8NwKQedRnwc+CGHusjrQ8oA4rD5RzgGeCMqOvq5b/rh9g7ZhJ5bUAeUBC3/BxBCKdCbc8Ac8Pla8KaIq8rrr57gItT5W8gfP0YgjO5csP3vBP4YirU1uvfw0h8WY+GB3A6wdlM7wBXDeH7/oqgv7ODIO0vIeiTfJzglLvHe/wDuiqsYQ3hGRfh+iUEpwe+A9zI3lP7sgma5usIztiYHbfP34fr18X/ocS9fjxB0/VV9p4aeXrU9QELCE67fTV8z6vD9SnxucVt9yH2hknktRGMTaxi7ynVV6VQbQuBFeF/098TfEFFXlf4ei5QBxTFrUuV2q4lOJ36deAXBEGRErX1fGg6FRERSZrGTEREJGkKExERSZrCREREkqYwERGRpClMREQkaQoTSTlmdpWZvWFmr5rZK2Z2zAD3v8jMpg1wn0oze30A2z9kZsUD2P4sM7syXL7GzP5lgPXF73+Omc0b4P7fMrNTBrjPRjMrHcg+Mn6lR12ASDwzO5bgAsVF7t4WfpllDmD/NIKZVV9nGGcxcPfTB7j9A8ADgzmWmaX32P8cgkn/Vg/g+FcP5tgiiVLLRFJNOVDr7m0A7l7r4RQhZnZyOFHga2Z2u5llhes3mtnVZvYXgtlOlwB3h62aHDNbbGZPh5MfPhw3FcViM1tlZv9HcCXwfsys3Mz+HL7X62Z2QtwxS8MWzVvh5IWvm9ndZnaKmT1rZmvN7Ohw+4vM7MZe3v9SM3sxrON+M8sN199hZj80syeB73fvb2YfIJhD6gdhTQeb2Utx7zfHzPabNyx8v/Piar/WzF4KP8tDw/UlZvZI+BnfTNz8TGb2WTN7ITzmzRZMxLk0bD1mm1le2Jrsbw41GcMUJpJqHgFmmNnbZvYTM/sggJllA3cAn3L3Iwha1f8vbr9Wdz/e3e8iuNL6Mx5MFNkJ/DdwnrsvJrhZ2XfDfX4GfMndj+2nnr8juD3BQuBIglkCejoE+C+CK/cPDfc5HvgX4N8O8Pv+1t2XuvuRBDeNuiTutfcBp7j7P3evcPfnCFoo/+ruC939HWC3mS0MN7mY4HM6kFoPJoT8aVgnwDeBv3gwGeMDwEEAZnYY8CmCSSQXAl0En++L4XbfIbjHxl3unnBXoYwt6uaSlOLujWa2GDgB+DDw63Cs4GWCSe/eDje9k6A1cUP4/Nd9vOVc4HDgUTOD4AZIW82siGD+r6fD7X5BMNlfTy8Ct1swIebv3b23MNng7q8BmNkbwOPu7mb2GsG8bP053My+Q3CzqHyC+0t0u9fduw6wP8CtwMVm9hWCL/2jE9ine1LPlcAnwuUTu5fd/X/MbGe4/mSCO/W9GH6GOeydXPBbBJ9RK/ClBI4rY5TCRFJO+AX6FPBU+IV8Ib23COI19bHegDd6tj7CwfMDziXk7n+2YBr3jwG/MLMfuPvPe2zWFrcci3se48B/Y3cA57j7KjO7iGC+r259/U493U/QqngCWOnudQns011jV48ae/tMDLjT3b/ey2uTCEIwg2Cep0RrljFG3VySUsxsrpnNiVu1ENhEMNldpZkdEq6/AHi65/6hBoLbEEMw4V1ZOLCPmWWY2XwPprbfbWbHh9t9po96ZhLcv+QW4DaCqdOHUgFBSymjrxp6Ef/74e6tBC2anxJ03Q3Wn7trMLPTCCZjhGAywfPMbHL42qTwcwFYDnwDuBv4fhLHllFOYSKpJh+408xWm9mrBPe1vib8wrwYuDdsrcSAm/p4jzuAmyy4U2MacB7BIPYqghbOB8LtLgZ+HA7At/TxXh8CXjGzl4FzCcZGhtI3CO5u+ShBYCbiHuBfw4Hyg8N1dxPeZTGJWq4FTgwH9D9KcIc/3H018O8Ed3B8Nay13Mw+B3S6+y+B7wFLzeykJI4vo5hmDRYZAyy4bqXI3b8RdS0yPmnMRGSUM7PfAQcDahVIZNQyERGRpGnMREREkqYwERGRpClMREQkaQoTERFJmsJERESS9v8B13wgn6CWVBsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(W_sparse_delta_sorted, label = 'delta')\n",
    "pyplot.ylabel('Similarity value')\n",
    "pyplot.xlabel('Sorted similarity index')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_sparse_CF has 7.41E+06 values and 10.62 % in common with CBF\n",
      "W_sparse_CBF has 3.10E+06 values and 25.38 % in common with CF\n",
      "W_sparse_delta has 7.87E+05 values\n"
     ]
    }
   ],
   "source": [
    "# How many values we have in common?\n",
    "\n",
    "print(\"W_sparse_CF has {:.2E} values and {:.2f} % in common with CBF\".format(W_sparse_CF.nnz, W_sparse_common.nnz/W_sparse_CF.nnz*100))\n",
    "print(\"W_sparse_CBF has {:.2E} values and {:.2f} % in common with CF\".format(W_sparse_CBF.nnz, W_sparse_common.nnz/W_sparse_CBF.nnz*100))\n",
    "\n",
    "print(\"W_sparse_delta has {:.2E} values\".format(W_sparse_delta.nnz))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What do we see?\n",
    "## The great majority of similarities are not common... the models are modeling two very different phenomena"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idea: \n",
    "\n",
    "## what if we were to weight features in such a way to approximate as well as possible the collaborative similarity?\n",
    "\n",
    "### Basic model:  $S_{CBF} = ICM \\cdot diag(feature\\_weights) \\cdot ICM^{T}$ \n",
    "\n",
    "\n",
    "### We want  $S_{CBF} = S_{CF}$\n",
    "\n",
    "### Hence we wish to find the weights such that  $S_{CF} = ICM \\cdot diag(feature\\_weights) \\cdot ICM^{T}$ \n",
    "\n",
    "### Now, we can rearrange the equations considering that we need only weights for features the two items have in common... this becomes a regression problem in form Ax = b. In our case A are the common features, x the weights and b the collaborative similarity\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Since it is a regression problem, we can use an already available solver like linalg in scipy.sparse\n",
    "\n",
    "### It takes as imput two data structures, one is sparse and the other is the target value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_delta = W_sparse_delta.tocoo()\n",
    "\n",
    "\n",
    "item_index_1 = W_sparse_delta.row[666]\n",
    "item_index_2 = W_sparse_delta.col[666]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item indices are 7 and 14\n"
     ]
    }
   ],
   "source": [
    "print(\"Item indices are {} and {}\".format(item_index_1, item_index_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative similarity value is 0.5295509099960327\n",
      "Content-based similarity value is 0.24285197257995605\n"
     ]
    }
   ],
   "source": [
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the common features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x10217 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 25 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_features = ICM_tags[item_index_1,:].multiply(ICM_tags[item_index_2,:])\n",
    "common_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  72,   73,  123,  138,  145,  146,  150,  246,  313,  511,  516,\n",
       "        561,  605,  714,  888,  918, 1155, 1514, 2091, 2357, 2462, 2463,\n",
       "       2579, 4573, 8075], dtype=int32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_features.indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## These items have just a common feature, that is a common scenario\n",
    "\n",
    "### Let's look for items having  a few features in common, say 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item indices are 1 and 11\n",
      "Collaborative similarity value is 0.3872489929199219\n",
      "Content-based similarity value is 0.10348083078861237\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([   8,  494,  678,  679, 2634], dtype=int32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_common_features = 5\n",
    "\n",
    "for inner_index in range(len(W_sparse_delta.row)):\n",
    "\n",
    "    item_index_1 = W_sparse_delta.row[inner_index]\n",
    "    item_index_2 = W_sparse_delta.col[inner_index]\n",
    "\n",
    "    common_features = ICM_tags[item_index_1,:].multiply(ICM_tags[item_index_2,:])\n",
    "    \n",
    "    if common_features.nnz >= min_common_features:\n",
    "        \n",
    "        # Bad programming, don't do this at home\n",
    "        break\n",
    "        \n",
    "        \n",
    "\n",
    "print(\"Item indices are {} and {}\".format(item_index_1, item_index_2))\n",
    "\n",
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))\n",
    "    \n",
    "common_features.indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import linalg\n",
    "\n",
    "loss_tolerance = 1e-6\n",
    "iteration_limit = 50000\n",
    "damp_coeff=0.0\n",
    "\n",
    "W_sparse_CF_value = W_sparse_CF[item_index_1, item_index_2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "linalg_result = linalg.lsqr(common_features, W_sparse_CF_value, show = False,\n",
    "                            atol=loss_tolerance, btol=loss_tolerance,\n",
    "                            iter_lim = iteration_limit, damp=damp_coeff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_weights = linalg_result[0].copy()\n",
    "feature_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.38969983e-04, 4.84242832e-05, 3.05072984e-03, 3.05072984e-03,\n",
       "       1.45272850e-04])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_weights[common_features.indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note that all weights will be zeros except for the features those items have in common, nothing can be learned if a feature is not common\n",
    "\n",
    "### If a dataset has a very sparse ICM, this FW method will struggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.38724899])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_CBF_value = common_features.dot(feature_weights)\n",
    "new_CBF_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative similarity value is 0.3872489929199219\n",
      "Content-based similarity value is 0.10348083078861237\n",
      "Weighted Content-based similarity value is 0.38724899291992176\n"
     ]
    }
   ],
   "source": [
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))\n",
    "\n",
    "print(\"Weighted Content-based similarity value is {}\".format(new_CBF_value[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note that we learned the CF similarity amost exactly...\n",
    "\n",
    "### Warning: 5 common features and 1 similarity value, more parameters than data points, overfitting!\n",
    "\n",
    "### if we want to apply this method to the whole dataset we have to compute the common features of all items having a common similarity value. Whe already know who they are, thanks to W_sparse_delta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from FeatureWeighting.CFW_D_Similarity_Linalg import CFW_D_Similarity_Linalg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 14888.50 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.73E-02, nonzero cells 1976311\n",
      "CFW_D_Similarity_Linalg: Content S structure has 566789 out of 1976311 ( 28.68%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 7.51E+04, average is: 1.32E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 16694.89 column/sec, elapsed time 0.01 min\n"
     ]
    }
   ],
   "source": [
    "CFW_weithing = CFW_D_Similarity_Linalg(URM_train, ICM_tags, W_sparse_CF)\n",
    "CFW_weithing.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Base.Evaluation.Evaluator import EvaluatorHoldout\n",
    "\n",
    "evaluator_validation = EvaluatorHoldout(URM_validation, cutoff_list=[5])\n",
    "evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[5])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 34000 ( 48.71% ) in 30.23 sec. Users per second: 1125\n",
      "EvaluatorHoldout: Processed 68000 ( 97.42% ) in 1.01 min. Users per second: 1122\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 1.04 min. Users per second: 1120\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.43311749087922413,\n",
       "  'PRECISION': 0.3117099306630827,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.31834851870960196,\n",
       "  'RECALL': 0.10381715744556805,\n",
       "  'MAP': 0.2338118477769968,\n",
       "  'MRR': 0.5153410501785864,\n",
       "  'NDCG': 0.15113096399648668,\n",
       "  'F1': 0.15575802336397737,\n",
       "  'HIT_RATE': 1.5585496533149963,\n",
       "  'ARHR': 0.7722000171909487,\n",
       "  'NOVELTY': 0.004278369748676905,\n",
       "  'AVERAGE_POPULARITY': 0.5617964021167546,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.9296870686082285,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9859347500144983,\n",
       "  'COVERAGE_ITEM': 0.13481883718752927,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.07564834753300252,\n",
       "  'COVERAGE_USER': 0.9989410114771459,\n",
       "  'COVERAGE_USER_CORRECT': 0.7273104553650648,\n",
       "  'DIVERSITY_GINI': 0.009607359053473404,\n",
       "  'SHANNON_ENTROPY': 6.957441386591916}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict, _ = evaluator_test.evaluateRecommender(itemKNNCF)\n",
    "results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 38000 ( 54.44% ) in 30.72 sec. Users per second: 1237\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 56.76 sec. Users per second: 1230\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.2667779305865951,\n",
       "  'PRECISION': 0.13209558191505644,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.1335546673542885,\n",
       "  'RECALL': 0.03252643425271064,\n",
       "  'MAP': 0.0897336669659144,\n",
       "  'MRR': 0.2831917464137816,\n",
       "  'NDCG': 0.062344355605169204,\n",
       "  'F1': 0.05219955823958507,\n",
       "  'HIT_RATE': 0.6604779095753825,\n",
       "  'ARHR': 0.3537051362863675,\n",
       "  'NOVELTY': 0.004327740319201193,\n",
       "  'AVERAGE_POPULARITY': 0.549577191560567,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.7136408482930834,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9427261249596072,\n",
       "  'COVERAGE_ITEM': 0.03941578503885404,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.018069469150828573,\n",
       "  'COVERAGE_USER': 0.9989410114771459,\n",
       "  'COVERAGE_USER_CORRECT': 0.43201007470162284,\n",
       "  'DIVERSITY_GINI': 0.0024442123987768096,\n",
       "  'SHANNON_ENTROPY': 4.877963904314825}}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict, _ = evaluator_test.evaluateRecommender(itemKNNCBF)\n",
    "results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 42000 ( 60.17% ) in 30.59 sec. Users per second: 1373\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 50.81 sec. Users per second: 1374\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.23673905984375296,\n",
       "  'PRECISION': 0.1299151911065066,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.13094879949571223,\n",
       "  'RECALL': 0.02882869501845794,\n",
       "  'MAP': 0.08527318555447344,\n",
       "  'MRR': 0.2554180276202147,\n",
       "  'NDCG': 0.055955297916752474,\n",
       "  'F1': 0.047186515513748195,\n",
       "  'HIT_RATE': 0.6495759555326343,\n",
       "  'ARHR': 0.3278255305331231,\n",
       "  'NOVELTY': 0.004455054445241026,\n",
       "  'AVERAGE_POPULARITY': 0.5004756900966632,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.7355381939417537,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9471055313498259,\n",
       "  'COVERAGE_ITEM': 0.05926411384701807,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.018631214305776612,\n",
       "  'COVERAGE_USER': 0.9989410114771459,\n",
       "  'COVERAGE_USER_CORRECT': 0.41133117719453904,\n",
       "  'DIVERSITY_GINI': 0.0025451980768019015,\n",
       "  'SHANNON_ENTROPY': 4.949569220480511}}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict, _ = evaluator_test.evaluateRecommender(CFW_weithing)\n",
    "results_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This algorithm has hyperparameters too:\n",
    "* topK, the usual number of neighbors\n",
    "* add_zeros_quota, the percentage of zero collaborative similarities to add to the train set, this helps the regressor in finding non-relevant features\n",
    "* normalize_similarity, apply or not the usual cosine denominator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ParameterTuning.SearchBayesianSkopt import SearchBayesianSkopt\n",
    "from skopt.space import Real, Integer, Categorical\n",
    "from ParameterTuning.SearchAbstractClass import SearchInputRecommenderArgs\n",
    "\n",
    "\n",
    "def hyperparameter_optimization_CFW_D(URM_train, ICM_train, W_sparse_CF):\n",
    "\n",
    "    recommender_class = CFW_D_Similarity_Linalg\n",
    "\n",
    "    parameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                     evaluator_validation=evaluator_validation,\n",
    "                                     evaluator_test=evaluator_test)\n",
    "\n",
    "\n",
    "    hyperparameters_range_dictionary = {}\n",
    "    hyperparameters_range_dictionary[\"topK\"] = Integer(5, 1000)\n",
    "    hyperparameters_range_dictionary[\"add_zeros_quota\"] = Real(low = 0, high = 1, prior = 'uniform')\n",
    "    hyperparameters_range_dictionary[\"normalize_similarity\"] = Categorical([True, False])\n",
    "\n",
    "\n",
    "    recommender_input_args = SearchInputRecommenderArgs(\n",
    "        CONSTRUCTOR_POSITIONAL_ARGS = [URM_train, ICM_train, W_sparse_CF],\n",
    "        CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "        FIT_POSITIONAL_ARGS = [],\n",
    "        FIT_KEYWORD_ARGS = {}\n",
    "    )\n",
    "\n",
    "\n",
    "    output_folder_path = \"result_experiments/\"\n",
    "\n",
    "    import os\n",
    "\n",
    "    # If directory does not exist, create\n",
    "    if not os.path.exists(output_folder_path):\n",
    "        os.makedirs(output_folder_path)\n",
    "\n",
    "    n_cases = 10\n",
    "    metric_to_optimize = \"MAP\"\n",
    "\n",
    "\n",
    "    # Clone data structure to perform the fitting with the best hyperparameters on train + validation data\n",
    "    recommender_input_args_last_test = recommender_input_args.copy()\n",
    "    recommender_input_args_last_test.CONSTRUCTOR_POSITIONAL_ARGS[0] = URM_train + URM_validation\n",
    "\n",
    "\n",
    "    parameterSearch.search(recommender_input_args,\n",
    "                           recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                           parameter_search_space = hyperparameters_range_dictionary,\n",
    "                           n_cases = n_cases,\n",
    "                           n_random_starts = int(n_cases/3),\n",
    "                           save_model = \"no\",\n",
    "                           output_folder_path = output_folder_path,\n",
    "                           output_file_name_root = recommender_class.RECOMMENDER_NAME,\n",
    "                           metric_to_optimize = metric_to_optimize\n",
    "                          )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 738, 'add_zeros_quota': 0.19340815501997705, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 14083.22 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.75E-02, nonzero cells 4275326\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 1288344 ( 35.96 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1031048 out of 4275326 ( 24.12%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.33E+05, average is: 1.29E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 14284.63 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 36000 ( 53.34% ) in 30.36 sec. Users per second: 1186\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 56.50 sec. Users per second: 1194\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 738, 'add_zeros_quota': 0.19340815501997705, 'normalize_similarity': False} - results: ROC_AUC: 0.1147172, PRECISION: 0.0478611, PRECISION_RECALL_MIN_DEN: 0.0536419, RECALL: 0.0262032, MAP: 0.0307601, MRR: 0.1133643, NDCG: 0.0372975, F1: 0.0338656, HIT_RATE: 0.2393053, ARHR: 0.1246850, NOVELTY: 0.0044362, AVERAGE_POPULARITY: 0.5151070, DIVERSITY_MEAN_INTER_LIST: 0.6181416, DIVERSITY_HERFINDAHL: 0.9236265, COVERAGE_ITEM: 0.0339856, COVERAGE_ITEM_CORRECT: 0.0090815, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.1930937, DIVERSITY_GINI: 0.0015152, SHANNON_ENTROPY: 4.2515738, \n",
      "\n",
      "EvaluatorHoldout: Processed 35000 ( 50.14% ) in 30.02 sec. Users per second: 1166\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 59.23 sec. Users per second: 1179\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 738, 'add_zeros_quota': 0.19340815501997705, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.2246421, PRECISION: 0.1154633, PRECISION_RECALL_MIN_DEN: 0.1165330, RECALL: 0.0262936, MAP: 0.0758153, MRR: 0.2412144, NDCG: 0.0522114, F1: 0.0428331, HIT_RATE: 0.5773165, ARHR: 0.2998450, NOVELTY: 0.0044365, AVERAGE_POPULARITY: 0.5151237, DIVERSITY_MEAN_INTER_LIST: 0.6191762, DIVERSITY_HERFINDAHL: 0.9238335, COVERAGE_ITEM: 0.0349218, COVERAGE_ITEM_CORRECT: 0.0125456, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.3813647, DIVERSITY_GINI: 0.0015310, SHANNON_ENTROPY: 4.2620133, \n",
      "\n",
      "\n",
      "DataIO: Json dumps supports only 'str' as dictionary keys. Transforming keys to string, note that this will alter the mapper content.\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 448.1565\n",
      "Function value obtained: -0.0308\n",
      "Current minimum: -0.0308\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 295, 'add_zeros_quota': 0.6174506263210966, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 15567.48 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.71E-02, nonzero cells 1946465\n",
      "CFW_D_Similarity_Linalg: Content S structure has 559634 out of 1946465 ( 28.75%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.09E+07, average is: 3.73E+01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 11175.65 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 46000 ( 68.16% ) in 30.44 sec. Users per second: 1511\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 44.94 sec. Users per second: 1502\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 295, 'add_zeros_quota': 0.6174506263210966, 'normalize_similarity': True} - results: ROC_AUC: 0.0051220, PRECISION: 0.0018907, PRECISION_RECALL_MIN_DEN: 0.0021347, RECALL: 0.0009764, MAP: 0.0010670, MRR: 0.0047708, NDCG: 0.0008292, F1: 0.0012877, HIT_RATE: 0.0094537, ARHR: 0.0048076, NOVELTY: 0.0070330, AVERAGE_POPULARITY: 0.0204508, DIVERSITY_MEAN_INTER_LIST: 0.7752774, DIVERSITY_HERFINDAHL: 0.9550532, COVERAGE_ITEM: 0.0939051, COVERAGE_ITEM_CORRECT: 0.0101114, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0090157, DIVERSITY_GINI: 0.0051867, SHANNON_ENTROPY: 5.6482931, \n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 365.9481\n",
      "Function value obtained: -0.0011\n",
      "Current minimum: -0.0308\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 398, 'add_zeros_quota': 0.38443825905322837, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 14039.62 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.23E-02, nonzero cells 2543376\n",
      "CFW_D_Similarity_Linalg: Content S structure has 695203 out of 2543376 ( 27.33%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 9.13E+04, average is: 1.31E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 13965.17 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 38000 ( 56.31% ) in 30.40 sec. Users per second: 1250\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 52.87 sec. Users per second: 1276\n",
      "SearchBayesianSkopt: New best config found. Config 2: {'topK': 398, 'add_zeros_quota': 0.38443825905322837, 'normalize_similarity': False} - results: ROC_AUC: 0.1297361, PRECISION: 0.0554062, PRECISION_RECALL_MIN_DEN: 0.0618623, RECALL: 0.0300236, MAP: 0.0356707, MRR: 0.1282101, NDCG: 0.0424462, F1: 0.0389441, HIT_RATE: 0.2770311, ARHR: 0.1434084, NOVELTY: 0.0043458, AVERAGE_POPULARITY: 0.5524141, DIVERSITY_MEAN_INTER_LIST: 0.6870370, DIVERSITY_HERFINDAHL: 0.9374054, COVERAGE_ITEM: 0.0457822, COVERAGE_ITEM_CORRECT: 0.0124520, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2174075, DIVERSITY_GINI: 0.0019747, SHANNON_ENTROPY: 4.6202129, \n",
      "\n",
      "EvaluatorHoldout: Processed 39000 ( 55.87% ) in 30.07 sec. Users per second: 1297\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 56.20 sec. Users per second: 1242\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 398, 'add_zeros_quota': 0.38443825905322837, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.2483872, PRECISION: 0.1338863, PRECISION_RECALL_MIN_DEN: 0.1350195, RECALL: 0.0300150, MAP: 0.0902459, MRR: 0.2689146, NDCG: 0.0591692, F1: 0.0490368, HIT_RATE: 0.6694316, ARHR: 0.3455664, NOVELTY: 0.0043460, AVERAGE_POPULARITY: 0.5524019, DIVERSITY_MEAN_INTER_LIST: 0.6881645, DIVERSITY_HERFINDAHL: 0.9376309, COVERAGE_ITEM: 0.0467185, COVERAGE_ITEM_CORRECT: 0.0154480, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.4186725, DIVERSITY_GINI: 0.0020015, SHANNON_ENTROPY: 4.6339741, \n",
      "\n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 380.7363\n",
      "Function value obtained: -0.0357\n",
      "Current minimum: -0.0357\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 16054.40 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.37E-03, nonzero cells 156468\n",
      "CFW_D_Similarity_Linalg: Content S structure has 58939 out of 156468 ( 37.67%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 8.36E+03, average is: 1.42E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 17225.65 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 53000 ( 78.53% ) in 30.36 sec. Users per second: 1746\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 38.42 sec. Users per second: 1757\n",
      "SearchBayesianSkopt: New best config found. Config 3: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False} - results: ROC_AUC: 0.1708601, PRECISION: 0.0748352, PRECISION_RECALL_MIN_DEN: 0.0839426, RECALL: 0.0418051, MAP: 0.0497524, MRR: 0.1694729, NDCG: 0.0594688, F1: 0.0536434, HIT_RATE: 0.3741758, ARHR: 0.1943579, NOVELTY: 0.0043615, AVERAGE_POPULARITY: 0.5319102, DIVERSITY_MEAN_INTER_LIST: 0.9067758, DIVERSITY_HERFINDAHL: 0.9813525, COVERAGE_ITEM: 0.0978373, COVERAGE_ITEM_CORRECT: 0.0291171, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2805175, DIVERSITY_GINI: 0.0073059, SHANNON_ENTROPY: 6.5358250, \n",
      "\n",
      "EvaluatorHoldout: Processed 52000 ( 74.49% ) in 30.06 sec. Users per second: 1730\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 40.23 sec. Users per second: 1735\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.3083873, PRECISION: 0.1804940, PRECISION_RECALL_MIN_DEN: 0.1822625, RECALL: 0.0417864, MAP: 0.1287235, MRR: 0.3406259, NDCG: 0.0822566, F1: 0.0678619, HIT_RATE: 0.9024698, ARHR: 0.4646329, NOVELTY: 0.0043640, AVERAGE_POPULARITY: 0.5309032, DIVERSITY_MEAN_INTER_LIST: 0.9076154, DIVERSITY_HERFINDAHL: 0.9815205, COVERAGE_ITEM: 0.0995225, COVERAGE_ITEM_CORRECT: 0.0397903, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.5032485, DIVERSITY_GINI: 0.0074347, SHANNON_ENTROPY: 6.5561975, \n",
      "\n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 124.0652\n",
      "Function value obtained: -0.0498\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 975, 'add_zeros_quota': 0.9992090128219687, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 12882.15 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.64E-02, nonzero cells 5298598\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2540392 ( 95.85 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1200019 out of 5298598 ( 22.65%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.54E+05, average is: 1.29E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 12711.66 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 32000 ( 47.42% ) in 30.12 sec. Users per second: 1063\n",
      "EvaluatorHoldout: Processed 64000 ( 94.83% ) in 1.01 min. Users per second: 1059\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 1.06 min. Users per second: 1060\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 975, 'add_zeros_quota': 0.9992090128219687, 'normalize_similarity': False} - results: ROC_AUC: 0.1260675, PRECISION: 0.0539778, PRECISION_RECALL_MIN_DEN: 0.0605504, RECALL: 0.0298846, MAP: 0.0346608, MRR: 0.1247746, NDCG: 0.0415482, F1: 0.0384703, HIT_RATE: 0.2698890, ARHR: 0.1389971, NOVELTY: 0.0042986, AVERAGE_POPULARITY: 0.5818300, DIVERSITY_MEAN_INTER_LIST: 0.5993590, DIVERSITY_HERFINDAHL: 0.9198700, COVERAGE_ITEM: 0.0305215, COVERAGE_ITEM_CORRECT: 0.0086134, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2138584, DIVERSITY_GINI: 0.0013637, SHANNON_ENTROPY: 4.1284689, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 754.5678\n",
      "Function value obtained: -0.0347\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 17229.18 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.31E-04, nonzero cells 37722\n",
      "CFW_D_Similarity_Linalg: Content S structure has 14953 out of 37722 ( 39.64%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.18E+03, average is: 1.46E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 18425.74 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 64000 ( 94.83% ) in 30.48 sec. Users per second: 2100\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 32.18 sec. Users per second: 2097\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False} - results: ROC_AUC: 0.1197082, PRECISION: 0.0516159, PRECISION_RECALL_MIN_DEN: 0.0587456, RECALL: 0.0307691, MAP: 0.0316900, MRR: 0.1149661, NDCG: 0.0397635, F1: 0.0385549, HIT_RATE: 0.2580793, ARHR: 0.1264367, NOVELTY: 0.0047119, AVERAGE_POPULARITY: 0.3827555, DIVERSITY_MEAN_INTER_LIST: 0.9483734, DIVERSITY_HERFINDAHL: 0.9896719, COVERAGE_ITEM: 0.1200262, COVERAGE_ITEM_CORRECT: 0.0338920, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2105241, DIVERSITY_GINI: 0.0118320, SHANNON_ENTROPY: 7.3153614, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 48.8650\n",
      "Function value obtained: -0.0317\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ferra\\Anaconda3\\envs\\RecSysCourse\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 16122.99 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.31E-04, nonzero cells 37722\n",
      "CFW_D_Similarity_Linalg: Content S structure has 14953 out of 37722 ( 39.64%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.18E+03, average is: 1.46E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 18427.29 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 59000 ( 87.42% ) in 30.12 sec. Users per second: 1959\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 34.22 sec. Users per second: 1972\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False} - results: ROC_AUC: 0.1197082, PRECISION: 0.0516159, PRECISION_RECALL_MIN_DEN: 0.0587456, RECALL: 0.0307691, MAP: 0.0316900, MRR: 0.1149661, NDCG: 0.0397635, F1: 0.0385549, HIT_RATE: 0.2580793, ARHR: 0.1264367, NOVELTY: 0.0047119, AVERAGE_POPULARITY: 0.3827555, DIVERSITY_MEAN_INTER_LIST: 0.9483734, DIVERSITY_HERFINDAHL: 0.9896719, COVERAGE_ITEM: 0.1200262, COVERAGE_ITEM_CORRECT: 0.0338920, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2105241, DIVERSITY_GINI: 0.0118320, SHANNON_ENTROPY: 7.3153614, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 51.5599\n",
      "Function value obtained: -0.0317\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ferra\\Anaconda3\\envs\\RecSysCourse\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 15825.80 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.31E-04, nonzero cells 37722\n",
      "CFW_D_Similarity_Linalg: Content S structure has 14953 out of 37722 ( 39.64%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.18E+03, average is: 1.46E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 17506.34 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 62000 ( 91.87% ) in 30.20 sec. Users per second: 2053\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 32.93 sec. Users per second: 2050\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 5, 'add_zeros_quota': 1.0, 'normalize_similarity': False} - results: ROC_AUC: 0.1197082, PRECISION: 0.0516159, PRECISION_RECALL_MIN_DEN: 0.0587456, RECALL: 0.0307691, MAP: 0.0316900, MRR: 0.1149661, NDCG: 0.0397635, F1: 0.0385549, HIT_RATE: 0.2580793, ARHR: 0.1264367, NOVELTY: 0.0047119, AVERAGE_POPULARITY: 0.3827555, DIVERSITY_MEAN_INTER_LIST: 0.9483734, DIVERSITY_HERFINDAHL: 0.9896719, COVERAGE_ITEM: 0.1200262, COVERAGE_ITEM_CORRECT: 0.0338920, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2105241, DIVERSITY_GINI: 0.0118320, SHANNON_ENTROPY: 7.3153614, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 50.1201\n",
      "Function value obtained: -0.0317\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 109, 'add_zeros_quota': 0.7014081878487206, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 15259.65 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 6.74E-03, nonzero cells 768817\n",
      "CFW_D_Similarity_Linalg: Content S structure has 250935 out of 768817 ( 32.64%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 3.42E+04, average is: 1.36E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 16285.97 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 45000 ( 66.68% ) in 30.14 sec. Users per second: 1493\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 44.71 sec. Users per second: 1509\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 109, 'add_zeros_quota': 0.7014081878487206, 'normalize_similarity': False} - results: ROC_AUC: 0.1519280, PRECISION: 0.0666647, PRECISION_RECALL_MIN_DEN: 0.0742669, RECALL: 0.0361778, MAP: 0.0430913, MRR: 0.1500437, NDCG: 0.0511441, F1: 0.0469024, HIT_RATE: 0.3333235, ARHR: 0.1710989, NOVELTY: 0.0043132, AVERAGE_POPULARITY: 0.5640230, DIVERSITY_MEAN_INTER_LIST: 0.8004935, DIVERSITY_HERFINDAHL: 0.9600963, COVERAGE_ITEM: 0.0692819, COVERAGE_ITEM_CORRECT: 0.0195675, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.2537136, DIVERSITY_GINI: 0.0035258, SHANNON_ENTROPY: 5.4301284, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 187.6550\n",
      "Function value obtained: -0.0431\n",
      "Current minimum: -0.0498\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 558, 'add_zeros_quota': 0.9916479993848962, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 13871.39 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.99E-02, nonzero cells 3408074\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2487521 ( 145.37 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 870841 out of 3408074 ( 25.55%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.82E+07, average is: 3.24E+01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 9537.08 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 39000 ( 57.79% ) in 30.75 sec. Users per second: 1268\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 52.54 sec. Users per second: 1284\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 558, 'add_zeros_quota': 0.9916479993848962, 'normalize_similarity': True} - results: ROC_AUC: 0.0068087, PRECISION: 0.0026761, PRECISION_RECALL_MIN_DEN: 0.0029438, RECALL: 0.0012481, MAP: 0.0014039, MRR: 0.0063225, NDCG: 0.0010943, F1: 0.0017023, HIT_RATE: 0.0133804, ARHR: 0.0064274, NOVELTY: 0.0066951, AVERAGE_POPULARITY: 0.0304717, DIVERSITY_MEAN_INTER_LIST: 0.6987859, DIVERSITY_HERFINDAHL: 0.9397551, COVERAGE_ITEM: 0.0824829, COVERAGE_ITEM_CORRECT: 0.0093624, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0125648, DIVERSITY_GINI: 0.0038452, SHANNON_ENTROPY: 5.1445020, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 673.3635\n",
      "Function value obtained: -0.0014\n",
      "Current minimum: -0.0498\n",
      "SearchBayesianSkopt: Search complete. Best config is 3: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False}\n",
      "\n",
      "CFW_D_Similarity_Linalg: URM Detected 28 (0.26 %) cold items.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 16697.62 column/sec, elapsed time 0.01 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.37E-03, nonzero cells 156468\n",
      "CFW_D_Similarity_Linalg: Content S structure has 58939 out of 156468 ( 37.67%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 8.36E+03, average is: 1.42E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 16084.92 column/sec, elapsed time 0.01 min\n",
      "EvaluatorHoldout: Processed 52000 ( 74.49% ) in 30.24 sec. Users per second: 1720\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 40.37 sec. Users per second: 1729\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 21, 'add_zeros_quota': 0.9970204201303217, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.3402849, PRECISION: 0.2056329, PRECISION_RECALL_MIN_DEN: 0.2075151, RECALL: 0.0452813, MAP: 0.1592698, MRR: 0.3757609, NDCG: 0.0913245, F1: 0.0742193, HIT_RATE: 1.0281646, ARHR: 0.5402525, NOVELTY: 0.0043710, AVERAGE_POPULARITY: 0.5228334, DIVERSITY_MEAN_INTER_LIST: 0.9088161, DIVERSITY_HERFINDAHL: 0.9817606, COVERAGE_ITEM: 0.0960584, COVERAGE_ITEM_CORRECT: 0.0401648, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.5181745, DIVERSITY_GINI: 0.0072256, SHANNON_ENTROPY: 6.5345659, \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "hyperparameter_optimization_CFW_D(URM_train, ICM_tags, W_sparse_CF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The best solution has a MAP of 0.159, better than the pure CBF\n",
    "\n",
    "### Let's try another ICM, the genres of the movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 9564.49 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.80E-02, nonzero cells 3199455\n",
      "CFW_D_Similarity_Linalg: Content S structure has 410297 out of 3199455 ( 12.82%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 3.95E+04, average is: 9.63E-02, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 10001.63 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 52000 ( 74.49% ) in 30.35 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 40.77 sec. Users per second: 1712\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.02916499532022997,\n",
       "  'PRECISION': 0.014420377055757095,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.014520180314405115,\n",
       "  'RECALL': 0.0027071735335412205,\n",
       "  'MAP': 0.006545446297251523,\n",
       "  'MRR': 0.02858097912249547,\n",
       "  'NDCG': 0.0033676185970134987,\n",
       "  'F1': 0.004558557618089629,\n",
       "  'HIT_RATE': 0.07210188527878059,\n",
       "  'ARHR': 0.030465541611751086,\n",
       "  'NOVELTY': 0.006578383126401176,\n",
       "  'AVERAGE_POPULARITY': 0.06496611868265492,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.9513712398493762,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9902715221339924,\n",
       "  'COVERAGE_ITEM': 0.2863964048310083,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.03913491246138002,\n",
       "  'COVERAGE_USER': 0.9989410114771459,\n",
       "  'COVERAGE_USER_CORRECT': 0.0648702023526718,\n",
       "  'DIVERSITY_GINI': 0.028092501784572593,\n",
       "  'SHANNON_ENTROPY': 8.16199141786077}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CFW_weithing_genre = CFW_D_Similarity_Linalg(URM_train, ICM_genres, W_sparse_CF)\n",
    "CFW_weithing_genre.fit()\n",
    "\n",
    "results_dict, _ = evaluator_test.evaluateRecommender(CFW_weithing_genre)\n",
    "results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 311, 'add_zeros_quota': 0.10513502946331879, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 9754.11 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.91E-02, nonzero cells 3315747\n",
      "CFW_D_Similarity_Linalg: Content S structure has 422479 out of 3315747 ( 12.74%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 4.06E+04, average is: 9.61E-02, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 9549.04 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 51000 ( 75.57% ) in 30.19 sec. Users per second: 1689\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 39.82 sec. Users per second: 1695\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 311, 'add_zeros_quota': 0.10513502946331879, 'normalize_similarity': False} - results: ROC_AUC: 0.0130754, PRECISION: 0.0062738, PRECISION_RECALL_MIN_DEN: 0.0069174, RECALL: 0.0031046, MAP: 0.0029256, MRR: 0.0124214, NDCG: 0.0029497, F1: 0.0041537, HIT_RATE: 0.0313690, ARHR: 0.0128146, NOVELTY: 0.0064127, AVERAGE_POPULARITY: 0.0722308, DIVERSITY_MEAN_INTER_LIST: 0.9472505, DIVERSITY_HERFINDAHL: 0.9894473, COVERAGE_ITEM: 0.2612115, COVERAGE_ITEM_CORRECT: 0.0232188, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0288217, DIVERSITY_GINI: 0.0234124, SHANNON_ENTROPY: 7.8786651, \n",
      "\n",
      "EvaluatorHoldout: Processed 49000 ( 70.20% ) in 30.04 sec. Users per second: 1631\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 42.30 sec. Users per second: 1650\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 311, 'add_zeros_quota': 0.10513502946331879, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0286970, PRECISION: 0.0146897, PRECISION_RECALL_MIN_DEN: 0.0148466, RECALL: 0.0031115, MAP: 0.0064701, MRR: 0.0279850, NDCG: 0.0038041, F1: 0.0051353, HIT_RATE: 0.0734485, ARHR: 0.0298954, NOVELTY: 0.0064128, AVERAGE_POPULARITY: 0.0723724, DIVERSITY_MEAN_INTER_LIST: 0.9479385, DIVERSITY_HERFINDAHL: 0.9895850, COVERAGE_ITEM: 0.2664545, COVERAGE_ITEM_CORRECT: 0.0340792, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0660437, DIVERSITY_GINI: 0.0237827, SHANNON_ENTROPY: 7.9009519, \n",
      "\n",
      "\n",
      "DataIO: Json dumps supports only 'str' as dictionary keys. Transforming keys to string, note that this will alter the mapper content.\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 105.8075\n",
      "Function value obtained: -0.0029\n",
      "Current minimum: -0.0029\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 733, 'add_zeros_quota': 0.3386570654013438, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 8794.61 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 6.72E-02, nonzero cells 7665992\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2123339 ( 37.08 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 862610 out of 7665992 ( 11.25%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.12E+05, average is: 2.46E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 6176.71 column/sec, elapsed time 0.03 min\n",
      "EvaluatorHoldout: Processed 40000 ( 59.27% ) in 30.34 sec. Users per second: 1318\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 50.69 sec. Users per second: 1331\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'topK': 733, 'add_zeros_quota': 0.3386570654013438, 'normalize_similarity': True} - results: ROC_AUC: 0.0151387, PRECISION: 0.0067806, PRECISION_RECALL_MIN_DEN: 0.0074503, RECALL: 0.0030645, MAP: 0.0033666, MRR: 0.0142440, NDCG: 0.0030298, F1: 0.0042212, HIT_RATE: 0.0339028, ARHR: 0.0147060, NOVELTY: 0.0066102, AVERAGE_POPULARITY: 0.0724974, DIVERSITY_MEAN_INTER_LIST: 0.9810454, DIVERSITY_HERFINDAHL: 0.9962062, COVERAGE_ITEM: 0.4531411, COVERAGE_ITEM_CORRECT: 0.0325812, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0310255, DIVERSITY_GINI: 0.0578154, SHANNON_ENTROPY: 9.3206447, \n",
      "\n",
      "EvaluatorHoldout: Processed 41000 ( 58.74% ) in 30.36 sec. Users per second: 1350\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 51.33 sec. Users per second: 1360\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 733, 'add_zeros_quota': 0.3386570654013438, 'normalize_similarity': True} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0330151, PRECISION: 0.0158472, PRECISION_RECALL_MIN_DEN: 0.0159845, RECALL: 0.0029810, MAP: 0.0075780, MRR: 0.0320944, NDCG: 0.0039486, F1: 0.0050180, HIT_RATE: 0.0792361, ARHR: 0.0346575, NOVELTY: 0.0066085, AVERAGE_POPULARITY: 0.0726897, DIVERSITY_MEAN_INTER_LIST: 0.9812930, DIVERSITY_HERFINDAHL: 0.9962558, COVERAGE_ITEM: 0.4567924, COVERAGE_ITEM_CORRECT: 0.0489655, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0696070, DIVERSITY_GINI: 0.0584840, SHANNON_ENTROPY: 9.3400308, \n",
      "\n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 155.7800\n",
      "Function value obtained: -0.0034\n",
      "Current minimum: -0.0034\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 142, 'add_zeros_quota': 0.21765604733227412, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 9258.07 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.33E-02, nonzero cells 1516446\n",
      "CFW_D_Similarity_Linalg: Content S structure has 209554 out of 1516446 ( 13.82%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 5.81E+04, average is: 2.77E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 7233.21 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 59000 ( 87.42% ) in 30.13 sec. Users per second: 1958\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 34.35 sec. Users per second: 1964\n",
      "SearchBayesianSkopt: New best config found. Config 2: {'topK': 142, 'add_zeros_quota': 0.21765604733227412, 'normalize_similarity': True} - results: ROC_AUC: 0.0209460, PRECISION: 0.0091455, PRECISION_RECALL_MIN_DEN: 0.0096527, RECALL: 0.0035217, MAP: 0.0045711, MRR: 0.0197361, NDCG: 0.0037909, F1: 0.0050852, HIT_RATE: 0.0457273, ARHR: 0.0206815, NOVELTY: 0.0063633, AVERAGE_POPULARITY: 0.0854621, DIVERSITY_MEAN_INTER_LIST: 0.9764181, DIVERSITY_HERFINDAHL: 0.9952807, COVERAGE_ITEM: 0.4541710, COVERAGE_ITEM_CORRECT: 0.0307087, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0410000, DIVERSITY_GINI: 0.0537483, SHANNON_ENTROPY: 9.1711103, \n",
      "\n",
      "EvaluatorHoldout: Processed 58000 ( 83.09% ) in 30.17 sec. Users per second: 1923\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 36.41 sec. Users per second: 1917\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 142, 'add_zeros_quota': 0.21765604733227412, 'normalize_similarity': True} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0450106, PRECISION: 0.0219013, PRECISION_RECALL_MIN_DEN: 0.0220226, RECALL: 0.0035593, MAP: 0.0109355, MRR: 0.0443301, NDCG: 0.0049489, F1: 0.0061235, HIT_RATE: 0.1095066, ARHR: 0.0490569, NOVELTY: 0.0063659, AVERAGE_POPULARITY: 0.0852008, DIVERSITY_MEAN_INTER_LIST: 0.9769219, DIVERSITY_HERFINDAHL: 0.9953816, COVERAGE_ITEM: 0.4615673, COVERAGE_ITEM_CORRECT: 0.0494336, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0925327, DIVERSITY_GINI: 0.0544877, SHANNON_ENTROPY: 9.1950034, \n",
      "\n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 85.9167\n",
      "Function value obtained: -0.0046\n",
      "Current minimum: -0.0046\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 994, 'add_zeros_quota': 0.9914471839151962, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 7664.34 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 9.02E-02, nonzero cells 10284772\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 4057703 ( 78.57 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1130295 out of 10284772 ( 10.99%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.06E+05, average is: 9.37E-02, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 9096.08 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 39000 ( 57.79% ) in 30.02 sec. Users per second: 1299\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 51.17 sec. Users per second: 1319\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 994, 'add_zeros_quota': 0.9914471839151962, 'normalize_similarity': False} - results: ROC_AUC: 0.0084535, PRECISION: 0.0041104, PRECISION_RECALL_MIN_DEN: 0.0045085, RECALL: 0.0018923, MAP: 0.0018082, MRR: 0.0079413, NDCG: 0.0015965, F1: 0.0025916, HIT_RATE: 0.0205521, ARHR: 0.0081216, NOVELTY: 0.0066073, AVERAGE_POPULARITY: 0.0577826, DIVERSITY_MEAN_INTER_LIST: 0.8795143, DIVERSITY_HERFINDAHL: 0.9759002, COVERAGE_ITEM: 0.1718004, COVERAGE_ITEM_CORRECT: 0.0147926, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0191906, DIVERSITY_GINI: 0.0107694, SHANNON_ENTROPY: 6.6325294, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 136.8538\n",
      "Function value obtained: -0.0018\n",
      "Current minimum: -0.0046\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 14, 'add_zeros_quota': 0.9884275648687755, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 10628.69 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.31E-03, nonzero cells 149520\n",
      "CFW_D_Similarity_Linalg: Content S structure has 22923 out of 149520 ( 15.33%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 7.27E+03, average is: 3.17E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 7653.69 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 27.38 sec. Users per second: 2465\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 14, 'add_zeros_quota': 0.9884275648687755, 'normalize_similarity': True} - results: ROC_AUC: 0.0160673, PRECISION: 0.0071866, PRECISION_RECALL_MIN_DEN: 0.0078000, RECALL: 0.0031577, MAP: 0.0035877, MRR: 0.0152350, NDCG: 0.0031300, F1: 0.0043875, HIT_RATE: 0.0359328, ARHR: 0.0159300, NOVELTY: 0.0065896, AVERAGE_POPULARITY: 0.0709215, DIVERSITY_MEAN_INTER_LIST: 0.9817950, DIVERSITY_HERFINDAHL: 0.9963561, COVERAGE_ITEM: 0.3262803, COVERAGE_ITEM_CORRECT: 0.0323940, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0323135, DIVERSITY_GINI: 0.0464914, SHANNON_ENTROPY: 9.1741055, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 33.8981\n",
      "Function value obtained: -0.0036\n",
      "Current minimum: -0.0046\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 11, 'add_zeros_quota': 0.0025526477649099326, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 10374.31 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.03E-03, nonzero cells 117480\n",
      "CFW_D_Similarity_Linalg: Content S structure has 17997 out of 117480 ( 15.32%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.87E+03, average is: 1.04E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 9933.59 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 28.37 sec. Users per second: 2379\n",
      "SearchBayesianSkopt: New best config found. Config 5: {'topK': 11, 'add_zeros_quota': 0.0025526477649099326, 'normalize_similarity': False} - results: ROC_AUC: 0.0214239, PRECISION: 0.0088580, PRECISION_RECALL_MIN_DEN: 0.0096078, RECALL: 0.0038324, MAP: 0.0046988, MRR: 0.0199955, NDCG: 0.0042951, F1: 0.0053501, HIT_RATE: 0.0442900, ARHR: 0.0208887, NOVELTY: 0.0065242, AVERAGE_POPULARITY: 0.0737152, DIVERSITY_MEAN_INTER_LIST: 0.9771148, DIVERSITY_HERFINDAHL: 0.9954201, COVERAGE_ITEM: 0.3001592, COVERAGE_ITEM_CORRECT: 0.0357644, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0396978, DIVERSITY_GINI: 0.0443297, SHANNON_ENTROPY: 9.0274463, \n",
      "\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 30.23 sec. Users per second: 2309\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 11, 'add_zeros_quota': 0.0025526477649099326, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0450893, PRECISION: 0.0205117, PRECISION_RECALL_MIN_DEN: 0.0206244, RECALL: 0.0037064, MAP: 0.0106282, MRR: 0.0433366, NDCG: 0.0055019, F1: 0.0062783, HIT_RATE: 0.1025586, ARHR: 0.0478015, NOVELTY: 0.0065249, AVERAGE_POPULARITY: 0.0738220, DIVERSITY_MEAN_INTER_LIST: 0.9772859, DIVERSITY_HERFINDAHL: 0.9954544, COVERAGE_ITEM: 0.3046531, COVERAGE_ITEM_CORRECT: 0.0494336, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0872091, DIVERSITY_GINI: 0.0446314, SHANNON_ENTROPY: 9.0370955, \n",
      "\n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 64.4853\n",
      "Function value obtained: -0.0047\n",
      "Current minimum: -0.0047\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 984, 'add_zeros_quota': 0.018205983460970358, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 8353.72 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 8.93E-02, nonzero cells 10184997\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 825747 ( 8.26 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1120416 out of 10184997 ( 11.00%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.69E+05, average is: 2.40E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 5825.42 column/sec, elapsed time 0.03 min\n",
      "EvaluatorHoldout: Processed 38000 ( 56.31% ) in 30.32 sec. Users per second: 1253\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 53.41 sec. Users per second: 1264\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 984, 'add_zeros_quota': 0.018205983460970358, 'normalize_similarity': True} - results: ROC_AUC: 0.0187283, PRECISION: 0.0083246, PRECISION_RECALL_MIN_DEN: 0.0089257, RECALL: 0.0034301, MAP: 0.0040561, MRR: 0.0175789, NDCG: 0.0036256, F1: 0.0048584, HIT_RATE: 0.0416228, ARHR: 0.0182509, NOVELTY: 0.0065275, AVERAGE_POPULARITY: 0.0837429, DIVERSITY_MEAN_INTER_LIST: 0.9762434, DIVERSITY_HERFINDAHL: 0.9952458, COVERAGE_ITEM: 0.6614549, COVERAGE_ITEM_CORRECT: 0.0386668, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0377086, DIVERSITY_GINI: 0.1022198, SHANNON_ENTROPY: 9.7377491, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 108.7568\n",
      "Function value obtained: -0.0041\n",
      "Current minimum: -0.0047\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'add_zeros_quota': 0.0, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity column 10681 ( 100 % ), 10222.89 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.68E-04, nonzero cells 53400\n",
      "CFW_D_Similarity_Linalg: Content S structure has 7976 out of 53400 ( 14.94%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.61E+03, average is: 3.28E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 7097.04 column/sec, elapsed time 0.03 min\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 28.24 sec. Users per second: 2390\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 5, 'add_zeros_quota': 0.0, 'normalize_similarity': True} - results: ROC_AUC: 0.0210090, PRECISION: 0.0083068, PRECISION_RECALL_MIN_DEN: 0.0090533, RECALL: 0.0037604, MAP: 0.0045966, MRR: 0.0194492, NDCG: 0.0042334, F1: 0.0051772, HIT_RATE: 0.0415339, ARHR: 0.0202829, NOVELTY: 0.0066238, AVERAGE_POPULARITY: 0.0797949, DIVERSITY_MEAN_INTER_LIST: 0.9820339, DIVERSITY_HERFINDAHL: 0.9964039, COVERAGE_ITEM: 0.4039884, COVERAGE_ITEM_CORRECT: 0.0323003, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0373937, DIVERSITY_GINI: 0.0489991, SHANNON_ENTROPY: 9.1904173, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 34.0706\n",
      "Function value obtained: -0.0046\n",
      "Current minimum: -0.0047\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 998, 'add_zeros_quota': 0.9867897848202423, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 8296.07 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 9.05E-02, nonzero cells 10324644\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 4049400 ( 77.92 %) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1134187 out of 10324644 ( 10.99%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.06E+05, average is: 9.37E-02, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 8383.81 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 38000 ( 56.31% ) in 30.34 sec. Users per second: 1253\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 53.62 sec. Users per second: 1259\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 998, 'add_zeros_quota': 0.9867897848202423, 'normalize_similarity': False} - results: ROC_AUC: 0.0083325, PRECISION: 0.0040186, PRECISION_RECALL_MIN_DEN: 0.0044292, RECALL: 0.0018746, MAP: 0.0017789, MRR: 0.0077956, NDCG: 0.0015853, F1: 0.0025566, HIT_RATE: 0.0200928, ARHR: 0.0079726, NOVELTY: 0.0066072, AVERAGE_POPULARITY: 0.0574816, DIVERSITY_MEAN_INTER_LIST: 0.8784679, DIVERSITY_HERFINDAHL: 0.9756910, COVERAGE_ITEM: 0.1703024, COVERAGE_ITEM_CORRECT: 0.0140436, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0187326, DIVERSITY_GINI: 0.0107106, SHANNON_ENTROPY: 6.6180073, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 141.5292\n",
      "Function value obtained: -0.0018\n",
      "Current minimum: -0.0047\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "CFW_D_Similarity_Linalg: URM Detected 49 (0.46 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 10432.60 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 9.36E-04, nonzero cells 106800\n",
      "CFW_D_Similarity_Linalg: Content S structure has 16316 out of 106800 ( 15.28%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.70E+03, average is: 1.04E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 10878.25 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 67487 ( 100.00% ) in 28.04 sec. Users per second: 2407\n",
      "SearchBayesianSkopt: New best config found. Config 9: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False} - results: ROC_AUC: 0.0227068, PRECISION: 0.0096670, PRECISION_RECALL_MIN_DEN: 0.0105141, RECALL: 0.0044033, MAP: 0.0050471, MRR: 0.0211947, NDCG: 0.0046920, F1: 0.0060506, HIT_RATE: 0.0483352, ARHR: 0.0223092, NOVELTY: 0.0063443, AVERAGE_POPULARITY: 0.0877874, DIVERSITY_MEAN_INTER_LIST: 0.9567825, DIVERSITY_HERFINDAHL: 0.9913537, COVERAGE_ITEM: 0.2595263, COVERAGE_ITEM_CORRECT: 0.0271510, COVERAGE_USER: 0.9657832, COVERAGE_USER_CORRECT: 0.0430465, DIVERSITY_GINI: 0.0275096, SHANNON_ENTROPY: 8.1607306, \n",
      "\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 29.17 sec. Users per second: 2393\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0489239, PRECISION: 0.0228354, PRECISION_RECALL_MIN_DEN: 0.0229485, RECALL: 0.0042091, MAP: 0.0119453, MRR: 0.0475742, NDCG: 0.0061491, F1: 0.0071081, HIT_RATE: 0.1141768, ARHR: 0.0530801, NOVELTY: 0.0063432, AVERAGE_POPULARITY: 0.0879623, DIVERSITY_MEAN_INTER_LIST: 0.9570249, DIVERSITY_HERFINDAHL: 0.9914022, COVERAGE_ITEM: 0.2637394, COVERAGE_ITEM_CORRECT: 0.0364198, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0953519, DIVERSITY_GINI: 0.0279432, SHANNON_ENTROPY: 8.1784491, \n",
      "\n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 63.0220\n",
      "Function value obtained: -0.0050\n",
      "Current minimum: -0.0050\n",
      "SearchBayesianSkopt: Search complete. Best config is 9: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False}\n",
      "\n",
      "CFW_D_Similarity_Linalg: URM Detected 28 (0.26 %) cold items.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 ( 100 % ), 10308.83 column/sec, elapsed time 0.02 min\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.49E-02, nonzero cells 7408876\n",
      "CFW_D_Similarity_Linalg: Content S density: 9.36E-04, nonzero cells 106800\n",
      "CFW_D_Similarity_Linalg: Content S structure has 16316 out of 106800 ( 15.28%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.70E+03, average is: 1.04E-01, average over all collaborative data is 8.65E-02\n",
      "Similarity column 10681 ( 100 % ), 11055.13 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 29.59 sec. Users per second: 2359\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 10, 'add_zeros_quota': 0.9868939699474233, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - ROC_AUC: 0.0510692, PRECISION: 0.0240932, PRECISION_RECALL_MIN_DEN: 0.0242147, RECALL: 0.0043749, MAP: 0.0128674, MRR: 0.0499678, NDCG: 0.0064915, F1: 0.0074052, HIT_RATE: 0.1204659, ARHR: 0.0564502, NOVELTY: 0.0063514, AVERAGE_POPULARITY: 0.0861611, DIVERSITY_MEAN_INTER_LIST: 0.9536432, DIVERSITY_HERFINDAHL: 0.9907259, COVERAGE_ITEM: 0.2520363, COVERAGE_ITEM_CORRECT: 0.0346410, COVERAGE_USER: 0.9989410, COVERAGE_USER_CORRECT: 0.0987865, DIVERSITY_GINI: 0.0260290, SHANNON_ENTROPY: 8.0660685, \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "hyperparameter_optimization_CFW_D(URM_train, ICM_genres, W_sparse_CF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here the best solution has a MAP of 0.0123... not great"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Since we are weighting features, and weights can be applied only to common features, we may expect a sparse ICM to have limited quality gains because the items will tend to have few features in common and so only few parameters that can be learned. A more dense ICM may exhibit much higher improvements due to more common features and model parameters, however, very different items may have the same features in common, choking the model.\n",
    "\n",
    "## Improvements of CF+CBF greatly depend on the ICM structure and collaborative data distribution...\n",
    "\n",
    "## Tip:\n",
    "### If your ICM is very sparse and you get poor results, you may try to aggregate features (e.g., put them in OR, add them...) to reduce their number and get a new denser ICM on which to try feature weighting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How about using these feature weights to perform feature selection?\n",
    "## Let's try to select the x% features with highest value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAftElEQVR4nO3deXxddZ3/8dcnW9Mm6Zambdq0pC2lULYCEQsIKIsWRgfH0RFcfoA6dePhOIoM/viNojOPhzo6PvT3UwarMjLKNgIOVVFENpWltEApbWmhtNCmTdt0SZNmT+7n98c5Kac3NzdLc3Nubt7Px+M+cs9yz/l8U7jvfM/yPebuiIiI9CUv7gJERCS7KShERCQtBYWIiKSloBARkbQUFCIiklZB3AVkwrRp07y6ujruMkRERo3nnntun7tXpFqWk0FRXV3NmjVr4i5DRGTUMLM3+lqmQ08iIpKWgkJERNJSUIiISFo5eY5CRGQs6+zspLa2lra2tl7LiouLqaqqorCwcMDbU1CIiOSY2tpaysrKqK6uxsyOzHd39u/fT21tLfPmzRvw9nToSUQkx7S1tVFeXn5USACYGeXl5Sl7GukoKEREclBySPQ3Px0FhYhIDvjjxj3c+sRrGdm2gkJEJAc8unkvP/nztoxsW0EhIpID3J08O3q6r/UGS0EhIpIDEgnIC88/FBcXs3///l6h0HPVU3Fx8aC2rctjRURyQCLSo6iqqqK2tpb6+vpe6/XcRzEYCgoRkRyQ8DevaCosLBzUfRL90aEnEZEc4O7kZegbXUEhIpIDgkNPg79HYiAUFCIiOSDhKChERKRvCXcylBMKChGRXOAOGcoJBYWISC5wdI5CRETSiN5wN9wUFCIiOUDnKEREJC1d9SQiImnphjsREUlLN9yJiEha0bGehlusQWFmy8xss5ltMbMbUyz/sJmtC19PmdnpcdQpIpLtEknPoxhOsQWFmeUDPwQuAxYDV5nZ4qTVtgEXuvtpwL8AK0a2ShGR0cFz9GT22cAWd9/q7h3A3cAV0RXc/Sl3PxhOPgMMbhB1EZExIuGek3dmzwZ2RKZrw3l9+Tjwu74WmtlyM1tjZmtSPaxDRCSXdSecvAwde4ozKFK1KOXDXM3sHQRB8U99bczdV7h7jbvXVFRUDFOJIiKjQ2tnNxOK8jOy7TiDohaYE5muAnYlr2RmpwE/Aa5w9/0jVJuIyKhyuL2LknGZeWhpnEGxGlhoZvPMrAi4ElgZXcHM5gL3Ax9191diqFFEZFRobu+itCgzQRHbM7PdvcvMrgMeAvKB29x9g5l9Klx+K/AVoBy4Jbw+uMvda+KqWUQkWzW3dzNhXGYOPcUWFADu/iDwYNK8WyPvPwF8YqTrEhEZTdyd5o4uSnPw0JOIiAyD1s5u3MnJcxQiIjIMDrd3AQoKERHpQ0t7NwAlOXh5rIiIDAP1KEREJK3mMCh0MltERFJq7lCPQkRE0mgOz1GUZug+CgWFiMgo13PoaUKG7sxWUIiIjHI6mS0iImk16/JYERFJ5439zRTl51GQn5mvdAWFiMgo19GdoKM7kbHtKyhEREa5gy0dnDF3csa2r6AQERnl6hramDmxOGPbV1CIiIxi7k5tQyuVk8ZnbB8KChGRUayhpZOOrgRVUxQUIiKSws6GVgBm6NCTiIik8sqeJgCOn16asX0oKERERrGXdh5iXEEe86aVZGwfCgoRkVHsic31nLOgnKKCzH2dKyhEREapHQda2LqvmQsWVmR0PwoKEZFRavXrBwBYOr88o/tRUIiIjFLrag8xoSifRTPLMrqfWIPCzJaZ2WYz22JmN6ZYfqKZPW1m7WZ2fRw1iohkI3fn0U17OXPuFPLzLKP7ii0ozCwf+CFwGbAYuMrMFietdgD4HPCdES5PRCSrrd/ZyPYDLbzn9MqM7yvOHsXZwBZ33+ruHcDdwBXRFdx9r7uvBjrjKFBEJFv98LEtlI0r4J2LZ2Z8X3EGxWxgR2S6Npw3JGa23MzWmNma+vr6Yy5ORCRbbd7dxO837Oaa86qZUlKU8f3FGRSpDqr5UDfm7ivcvcbdayoqMnupmIhInO5Y9QbFhXl87Lx5I7K/OIOiFpgTma4CdsVUi4jIqNDU1sm9z9Vy8UkzRqQ3AfEGxWpgoZnNM7Mi4EpgZYz1iIhkNXfnKw9soKWjm4+dVz1i+y0YsT0lcfcuM7sOeAjIB25z9w1m9qlw+a1mNhNYA0wEEmb2eWCxuzfGVbeISFzue34nv3phJ5+8YD5nHTd1xPYbW1AAuPuDwINJ826NvN9NcEhKRGRM276/hZtXbuCs46Zww7ITR3TfujNbRCTLbd/fwlU/fobuhPPvHzg94zfYJVNQiIhksbpDrVz142c43N7FPZ9cSnUGhxPvS6yHnkREpG9rdzTwmV88R2NbF3f9/VJOrZoUSx3qUYiIZBl35+dPv877bnkSgLuXxxcSoB6FiEhW2dnQyr/8eiO/37Cb8xdO4/tXnsHUEbpfoi8KChGRLODu3PtcLV95YAMJd770rkV8+sIF5I3wietUFBQiIjF7dU8TX125gade28/Z86by3b87naopE+Iu6wgFhYhITA40d/CjJ17jtie3UTKugK9fcTIfOnsuBfnZdfpYQSEiMsJaO7r52VOvc8vjW2hq6+L9Z1Vxw7JFTC8rjru0lBQUIiIjpL6pnTtWvcHtT73OwZZOLj5xOjcsOzHjjzI9VgoKEZEM6k44f3q1nrtWbeeRTXvpTjiXnDSdT164gLdUj9x4TcdCQSEiMswaWjr448t7eXLLPv70Sj37mzuYWlLEJ86fxwfOmsPx00vjLnFQFBQiIsNg96E2Hli7k0de3suaNw6QcCgvKeJtC6dx2SkzuejEGRQVZNdJ6oFSUIiIDIG7s21fM49trucPG3az+vUgHE6eNZHPvuN4LjlpBqdVTcIs/vsgjpWCQkRkANyd2oOt/GXLPv6yZR+rth5g3+F2ABbNKOMzbz+eD9RUcVz5yA/al2kKChGRFA61dvLijgbWhq91tQ3sO9wBwMyJxZy/cBpvqZ7KOQvKmRfDiK4jSUEhImNeV3eCTbubeGFHA2u3N7B2x0Feq28+svz46aW8fdF0Tp09ifOOL2dBRWlOHFIaKAWFiIwpbZ3dbNjVyMa6RjaGPzfVNdLelQBgWmkRS+ZM5m/OmM2SOVM4bc4kJhYXxlx1vBQUIpKT3J29Te28sqeJzbubWL/zEC/tPMS2fc0kPFhn0vhCFldO5CNLj+O0qkmcOXcKVVPGj6newkAoKERk1Gvt6GbDrkNsrGvk5bomXt3TxCt7mmhs6zqyTkXZOJbMmczlp1Zy8qxJnDJ7IrMnKxQGQkEhIqNGc3sX2/Y1s3VfM6/tPcyW+sNs2RP87A67CZMnFLJweinvOX0WJ8woY+GMUhbNKKO8dFzM1Y9eCgoRySqJhLOzofVIGGzdd5it9c1srW9md2PbkfXMYM6UCRw/vZR3njyDU2dP4tSqScycWKxewjCLNSjMbBnwfSAf+Im7fzNpuYXLLwdagGvc/fkRL1REhlVjWye7GlrZ1dDKzoOt1IY/t+w9zLZ9zUdOLAOUFRewoKKUc8OrjeZPK6F6WgnzppVQXJgfYyvGjtiCwszygR8ClwK1wGozW+nuGyOrXQYsDF9vBf4j/CkiWaq9q5udB1vZ3djGnsY2dh9qDwKhJxgaWmmKnDsAKMrPY+akYhZUlHD+wmnMDwNhfkUp00qL1EOIWZw9irOBLe6+FcDM7gauAKJBcQXwX+7uwDNmNtnMKt29buTLzSx3Z9/hDmoPtrD/cAfd7inW6fPTfWxzMGunW3+4tt9nAzJXyzDVOMhffco6s+3fo68PDLaextZOdhxsYceBVnYcbKHuUNuR8wU9Jo0vZPbk8VRNmcDS+eXMmlzMrMnjmTV5PFWTxzOtdFxWPPJTUoszKGYDOyLTtfTuLaRaZzbQKyjMbDmwHGDu3LnDWuhwa+vs5sUdDWysa2RrfTObdjeyeffRV2iIjCYVZeOYM2U8NcdNYc7UCVSXl1A5qZgZk4qZObGYknE6HTqaDehfz8y+5e7/1N+8QUr150Py3ywDWSeY6b4CWAFQU1Mz8D9dR8Dh9i5Wv36AF7Y38MzW/azd3kBHd3AMtqy4gBNmlPGe02exoKKU48onUFE2jvw+/rqylL+S4MTeoOYPdjupZ/e5fl+fSLV+39vu63cwuFoG29a+9P07PvY6h+vfo68Fw/fv3XvJ+MJ8xhfpXEEuG2jMXwokh8JlKeYNRi0wJzJdBewawjpZqamtk9+uq+O3L9XxzNb9dHY7eQYnz5rEtedVU1M9ldPnTKKidJyOv4pIVksbFGb2aeAzwHwzWxdZVAY8eYz7Xg0sNLN5wE7gSuBDSeusBK4Lz1+8FTiU7ecnuroT/PjP2/jJn7eyv7mD6vIJXHvePC48oYIlcyarCy4io05/31p3Ar8DvgHcGJnf5O4HjmXH7t5lZtcBDxFcHnubu28ws0+Fy28FHiS4NHYLweWx1x7LPjPJ3bnz2e3c+sRr7DjQytsXVfDpCxdw9ryp6jGIyKhmA70SJbycdQaRcHH37Rmq65jU1NT4mjVrRmx/jW2d3LxyA/c/v5OzjpvC358/j2WnVI7Y/kVEjpWZPefuNamWDfRk9nXAzcAeoOdOGAdOG44CR6tEwvn5M2/w7Yc2c7i9i+vecTxfuPQEXeYnIjlloAfMPw8scvf9mSxmtPnfv3qJu1fvoOa4Kfyfdy9myZzJcZckIjLsBhoUO4BDmSxktPn5069z9+odXHNuNf/87sV9Xs4qIjLa9XfV0xfCt1uBx83st0B7z3J3/24Ga8ta9z9fyz8/sIGLTpzOTX91kkJCRHJafz2KsvDn9vBVFL7GrIc37uFL967jLdVTuOXDZ1KYnxd3SSIiGZU2KNz9ayNVyGhwoLmD63/5IidVlvGf156tkStFZEwY6FVPv6b30BmHgDXAj9y9rfencs/dq7dzqLWTu/52KaW6cU5ExoiBHjfZChwGfhy+GgkulT0hnM55Oxta+cGjW7jwhAoWz5oYdzkiIiNmoH8Wn+HuF0Smf21mf3L3C8xsQyYKyyZd3Qk+e0fwvKR/fe8pMVcjIjKyBtqjqDCzI2N3h++nhZMdw15Vlnl22wHW7mjgpr86iTlTJ8RdjojIiBpoj+KLwF/M7DWCEYjnAZ8xsxLg9kwVly3ufHY7E4sLeN8ZVXGXIiIy4gYUFO7+oJktBE4kCIpNkRPY38tUcdlgb1Mbv1+/m6vPrdaY+yIyJvV3w91F7v6omb0vadF8M8Pd789gbVnhl2tq6Uo4H3prdj81T0QkU/rrUVwIPAq8J8UyB3I+KO5ctZ3zji9nQUVp3KWIiMSivxvuvhr+zNrnQGRS7cEWdja0svyC+XGXIiISmwFd9WRmM8zsp2b2u3B6sZl9PLOlxe9Pr+wDYOn88pgrERGJz0Avj/0ZwZPoZoXTrxAMPZ7TfvncDuZNK2HhdB12EpGxa6BBMc3d/5vwoUXu3gV0Z6yqLODubNzVyMUnTteDiERkTBtoUDSbWTnheE9mtpQcfz7Fq3sP096VYF5FSdyliIjEqr/LYz8PPAncADxAcFnsk0AF8IHMlxeflWt3kZ9nvOvkmXGXIiISq/4uj60Cvk9wo90m4GHgceAed9+X2dLi9fz2g5wyayLTSsfFXYqISKzSHnpy9+vd/VxgJnA9sAq4CFhnZhtHoL7YbNvXzAKdxBYRGfBYT+OBicCk8LULeClTRcWtqa2TukNtVJfr/ISISH/nKFYAJwNNBL2Jp4DvuvvBY9mpmU0F7gGqgdeBv0u1TTO7DXg3sNfdR2x87027mwCYq5FiRUT6veppLjAO2A3sBGqBhmHY743AI+6+EHgknE7lZ8CyYdjfoOxpDMY7PKlSDygSEenvHMUy4C3Ad8JZXwRWm9kfzOxYnqd9BW8OT3478N4+9v8n4MAx7GdI9ja2A1BRphPZIiL9nqNwdwfWm1kDwb0ThwgOB50NfHWI+53h7nXh9uvMbPoQt3OEmS0HlgPMnXtsI73WHWqluDCPKRMKj7UsEZFRr79zFJ8DzgXOAzoJ7ql4GriNfk5mm9kfCa6WSnbTkCrth7uvAFYA1NTU+LFsa3djOzMmFmOmO7JFRPrrUVQD9wL/2NMDGCh3v6SvZWa2x8wqw95EJbB3MNvOtIaWDqZMKIq7DBGRrNDfOYovuPu9gw2JAVgJXB2+v5rgru+s0dDSyWQddhIRAQY+1tNw+yZwqZm9ClwaTmNms8zswZ6VzOwugkNdi8ysdqSGNj/Q3MHUEvUoRERg4DfcDSt33w9cnGL+LuDyyPRVI1lXuE/2HW6nXEEhIgLE16PIWi0d3bR3JZhaoktjRURAQdHLgeYOAMpL1aMQEQEFRS/7e4JCh55ERAAFRS8Hw6CYoqAQEQEUFL00tIZBofsoREQABUUvDS2dAEwer/soRERAQdFLfVM7BXnGJAWFiAigoOilvqmdaaXjyMvTOE8iIqCg6KX+cLuGFxcRiVBQJGlu76KsOJYb1kVEspKCIklbZ4JxBfq1iIj00DdikrbObooL8+MuQ0QkaygokrR1datHISISoW/EJK0dCcYX6RyFiEgPBUWS1o4uJhTp0JOISA8FRYS709LZraAQEYlQUES0dyVwRyezRUQiFBQRrR3dAOpRiIhEKCgimtq6ACgr1jhPIiI9FBQRB1p6hhhXUIiI9FBQRDS3Bz2KknG6PFZEpIeCIqK9KzhHoZPZIiJviiUozGyqmT1sZq+GP6ekWGeOmT1mZi+b2QYz+4dM19XemQDQndkiIhFxfSPeCDzi7guBR8LpZF3AF939JGAp8FkzW5zJojq6FRQiIsni+ka8Arg9fH878N7kFdy9zt2fD983AS8DszNZVFtncOipSEEhInJEXN+IM9y9DoJAAKanW9nMqoEzgFWZLKolvI+iRGM9iYgckbFvRDP7IzAzxaKbBrmdUuA+4PPu3phmveXAcoC5c+cOZhdHdHU7AIXqUYiIHJGxoHD3S/paZmZ7zKzS3evMrBLY28d6hQQhcYe739/P/lYAKwBqamp8KDV3JoJzFAV6XraIyBFx/em8Erg6fH818EDyCmZmwE+Bl939uyNRVE+PQkEhIvKmuILim8ClZvYqcGk4jZnNMrMHw3XOAz4KXGRma8PX5Zksqiu86ilfQSEickQsZ23dfT9wcYr5u4DLw/d/AUb0G7sz4RTmG0FnRkREQHdmH6WrO0FBnn4lIiJR+laM6Ox2CvLVmxARiVJQRNQ3tVOYr1+JiEiUvhUjCvKNw+EzKUREJKCgiHCHWZOL4y5DRCSrKCgiut3J06WxIiJHUVBEdHe7brYTEUmioIjodidP91CIiBxFQRGRSLjuyhYRSaKgiOh2BYWISDIFRUS3ehQiIr0oKCK6E06+zlGIiBxFQRHRndDlsSIiyRQUEQlXj0JEJJmCIqIroUEBRUSSKSgiEgndRyEikkxBEaHLY0VEelNQRHQnUI9CRCSJgiLC3VGHQkTkaAqKCHdQh0JE5GgKigjHMZQUIiJRCooId8jTb0RE5Cj6WoxIuHoUIiLJYgkKM5tqZg+b2avhzykp1ik2s2fN7EUz22BmX8t0XQ4oJ0REjhZXj+JG4BF3Xwg8Ek4nawcucvfTgSXAMjNbmtGqXJfHiogkiysorgBuD9/fDrw3eQUPHA4nC8OXZ7Ko4NCTiIhExRUUM9y9DiD8OT3VSmaWb2Zrgb3Aw+6+qq8NmtlyM1tjZmvq6+uHVJSjy2NFRJIVZGrDZvZHYGaKRTcNdBvu3g0sMbPJwK/M7BR3X9/HuiuAFQA1NTVD6nm4Dj2JiPSSsaBw90v6WmZme8ys0t3rzKySoMeQblsNZvY4sAxIGRTDQYeeRER6i+vQ00rg6vD91cADySuYWUXYk8DMxgOXAJsyWZQ7uupJRCRJXEHxTeBSM3sVuDScxsxmmdmD4TqVwGNmtg5YTXCO4jeZLkyHnkREjpaxQ0/puPt+4OIU83cBl4fv1wFnjGRdOvQkItKb7syO0KCAIiK9KSgiHD3hTkQkmYIiIqEehYhILwqKCNdgTyIivSgojqIn3ImIJFNQROjQk4hIbwqKCNfzKEREelFQRDjo0JOISBIFRUQi4ZiOPYmIHEVBEZHRh12IiIxSCoooDTMuItKLgiIi4a6rnkREkigoIjTKuIhIbwqKCHfI02VPIiJHUVBEvOvkGZw4syzuMkREskosz6PIVt+7ckQffyEiMiqoRyEiImkpKEREJC0FhYiIpKWgEBGRtBQUIiKSloJCRETSUlCIiEhaCgoREUnL3HNvcG0zqwfeGOLHpwH7hrGcbDaW2gpqby4bS22FzLT3OHevSLUgJ4PiWJjZGnevibuOkTCW2gpqby4bS22FkW+vDj2JiEhaCgoREUlLQdHbirgLGEFjqa2g9uaysdRWGOH26hyFiIikpR6FiIikpaAQEZG0FBQhM1tmZpvNbIuZ3Rh3PUNhZnPM7DEze9nMNpjZP4Tzp5rZw2b2avhzSuQzXw7bvNnM3hWZf5aZvRQu+79mlrXPiDWzfDN7wcx+E07nbHvNbLKZ3Wtmm8J/53Nytb1m9o/hf8frzewuMyvOpbaa2W1mttfM1kfmDVv7zGycmd0Tzl9lZtVDLtbdx/wLyAdeA+YDRcCLwOK46xpCOyqBM8P3ZcArwGLg34Abw/k3At8K3y8O2zoOmBf+DvLDZc8C5wAG/A64LO72pWn3F4A7gd+E0znbXuB24BPh+yJgci62F5gNbAPGh9P/DVyTS20FLgDOBNZH5g1b+4DPALeG768E7hlyrXH/srLhFf6SH4pMfxn4ctx1DUO7HgAuBTYDleG8SmBzqnYCD4W/i0pgU2T+VcCP4m5PH22sAh4BLooERU62F5gYfnla0vyca28YFDuAqQSPbP4N8M5caytQnRQUw9a+nnXC9wUEd3LbUOrUoadAz3+UPWrDeaNW2M08A1gFzHD3OoDw5/Rwtb7aPTt8nzw/G30PuAFIROblanvnA/XAf4aH2n5iZiXkYHvdfSfwHWA7UAcccvc/kINtTTKc7TvyGXfvAg4B5UMpSkERSHXMctReN2xmpcB9wOfdvTHdqinmeZr5WcXM3g3sdffnBvqRFPNGTXsJ/io8E/gPdz8DaCY4PNGXUdve8Nj8FQSHWWYBJWb2kXQfSTFvVLR1gIbSvmFru4IiUAvMiUxXAbtiquWYmFkhQUjc4e73h7P3mFlluLwS2BvO76vdteH75PnZ5jzgr83sdeBu4CIz+wW5295aoNbdV4XT9xIERy629xJgm7vXu3sncD9wLrnZ1qjhbN+Rz5hZATAJODCUohQUgdXAQjObZ2ZFBCd+VsZc06CFVzv8FHjZ3b8bWbQSuDp8fzXBuYue+VeGV0fMAxYCz4Zd3iYzWxpu839FPpM13P3L7l7l7tUE/2aPuvtHyN327gZ2mNmicNbFwEZys73bgaVmNiGs8WLgZXKzrVHD2b7ott5P8P/H0HpTcZ/MyZYXcDnBVUKvATfFXc8Q2/A2gq7lOmBt+Lqc4LjkI8Cr4c+pkc/cFLZ5M5GrQYAaYH247AcM8STYCLb97bx5Mjtn2wssAdaE/8b/A0zJ1fYCXwM2hXX+nOCKn5xpK3AXwfmXToK//j8+nO0DioFfAlsIroyaP9RaNYSHiIikpUNPIiKSloJCRETSUlCIiEhaCgoREUlLQSEiImkpKCQnmdlN4cij68xsrZm9dZCfv8bMZg3yM9XRkUCTln07rOfbg9lm+NklZnb5YD8nMlwK4i5AZLiZ2TnAuwlG0m03s2kEI60O9PP5BCOVrmf47uL9JFDh7u1D+OwSgmvlHxzoB8Kbr8zdE/2uLNIP9SgkF1UC+3q+lN19n7vvAjCzi8MB9V4KnwcwLpz/upl9xcz+QjACZw1wR9gbGR+O+f+EmT1nZg9Fhlk4y8xeNLOngc+mKsbMVgIlwCoz+6CZVZjZfWa2OnydF653tpk9Fdb3lJktCkcK+DrwwbCWD5rZzWZ2fWT768PeTLUFz6i4BXgemGNmXwr3sc7MvhauX2Jmvw3rXm9mH8zAv4HkkrjvTtRLr+F+AaUEd6W/AtwCXBjOLyYYTfOEcPq/CAZOBHgduCGyjceBmvB9IfAUQY8A4IPAbeH7dZHtf5vIkNFJNR2OvL8TeFv4fi7BkCsQDCNeEL6/BLgvfH8N8IPI528Gro9MrycYrrqaYBTdpeH8dwIrCAaHyyMYqvsC4G+BH0c+PynufzO9svulQ0+Sc9z9sJmdBZwPvAO4x4KnFr5AMNDcK+GqtxP0Ar4XTt/TxyYXAacADwdHdMgH6sxsEjDZ3Z8I1/s5cNkASrwEWGxvPmhtopmVEQzadruZLSQYiqVwIO1N8oa7PxO+f2f4eiGcLiUYI+jPwHfM7FsEw578eQj7kTFEQSE5yd27CXoFj5vZSwSDo63t52PNfcw3YIO7n3PUTLPJDG3Y5jyCB8q0Jm3v/wGPufvfWPA8kcf7+HwXRx82Lo68j7bBgG+4+4+SNxAG6eXAN8zsD+7+9cE2QsYOnaOQnBMe218YmbUEeINggLlqMzs+nP9R4Inkz4eaCB4nC8EgbBXhSXLMrNDMTnb3BuCQmb0tXO/DAyzxD8B1kXqXhG8nATvD99f0UQsEh8nODD97JsEzG1J5CPiYBc8nwcxmm9n08GquFnf/BcHDgc4cYN0yRikoJBeVEhzC2Whm6wieN3yzu7cB1wK/DHsZCeDWPrbxM+BWM1tLcKjp/cC3zOxFgp7JueF61wI/DE9mt6baUAqfA2rCE8wbgU+F8/+N4C/8J8N99niM4FDV2vDE833A1LC2TxOci+nFgyfC3Qk8Hbb3XoLAORV4Nvz8TcC/DrBuGaM0eqyIiKSlHoWIiKSloBARkbQUFCIikpaCQkRE0lJQiIhIWgoKERFJS0EhIiJp/X/4Kr0ZOb1WdAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as pyplot\n",
    "%matplotlib inline  \n",
    "\n",
    "pyplot.plot(np.sort(CFW_weithing.D_best))\n",
    "pyplot.ylabel('Weight')\n",
    "pyplot.xlabel('Sorted features')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Something to think about... do we select features with the 'highest' weight, hence excluding the negative ones, or do we select those that have the 'highest absolute' value, accounting also for the strong negative ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected 3065/10217 features, weights: 3.39E-03 to 2.64E-01\n",
      "ItemKNNCBFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 10919.31 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 39000 ( 55.87% ) in 30.06 sec. Users per second: 1297\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 53.74 sec. Users per second: 1299\n",
      "Selected 5108/10217 features, weights: 0.00E+00 to 2.64E-01\n",
      "ItemKNNCBFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 10281.99 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 37000 ( 53.01% ) in 30.32 sec. Users per second: 1220\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 56.56 sec. Users per second: 1234\n",
      "Selected 7151/10217 features, weights: 0.00E+00 to 2.64E-01\n",
      "ItemKNNCBFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 8883.01 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 38000 ( 54.44% ) in 30.22 sec. Users per second: 1258\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 57.68 sec. Users per second: 1210\n",
      "Selected 9195/10217 features, weights: -1.33E-03 to 2.64E-01\n",
      "ItemKNNCBFRecommender: URM Detected 49 (0.46 %) cold items.\n",
      "Similarity column 10681 ( 100 % ), 9496.84 column/sec, elapsed time 0.02 min\n",
      "EvaluatorHoldout: Processed 35000 ( 50.14% ) in 30.30 sec. Users per second: 1155\n",
      "EvaluatorHoldout: Processed 69804 ( 100.00% ) in 59.82 sec. Users per second: 1167\n"
     ]
    }
   ],
   "source": [
    "argsort_features = np.argsort(-CFW_weithing.D_best)\n",
    "\n",
    "map_list = []\n",
    "selection_quota_list = [0.3, 0.5, 0.7, 0.9]\n",
    "\n",
    "for selection_quota in selection_quota_list:\n",
    "    \n",
    "    n_to_select = int(selection_quota*len(argsort_features))\n",
    "    selected_features = argsort_features[:n_to_select]\n",
    "    \n",
    "    ICM_selected = ICM_tags[:,selected_features]\n",
    "    \n",
    "    print(\"Selected {}/{} features, weights: {:.2E} to {:.2E}\".format(n_to_select, len(argsort_features),\n",
    "                                                                min(CFW_weithing.D_best[selected_features]),\n",
    "                                                                max(CFW_weithing.D_best[selected_features])))\n",
    "    \n",
    "    itemKNNCBF = ItemKNNCBFRecommender(URM_train, ICM_selected)\n",
    "    itemKNNCBF.fit(**best_params_ItemKNNCBF)\n",
    "    \n",
    "    results_dict, _ = evaluator_test.evaluateRecommender(itemKNNCBF)\n",
    "    map_list.append(results_dict[5][\"MAP\"])\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9dX48c/JThb2sCVAWBIUEBHCpoILYMWluK8I4kJxF+3zq12e9uny9LGtilIsioBAtW6tttRiZXOnLAHZlyQsQgAhgEBYsp/fH/dGhhjITJjJnSTn/XrNi5l7v3fu+TLJnHzv8j2iqhhjjDH+ivA6AGOMMXWLJQ5jjDEBscRhjDEmIJY4jDHGBMQShzHGmIBEeR1AbWjZsqWmpaV5HYYxxtQpK1as2K+qyZWXN4jEkZaWRlZWltdhGGNMnSIiX1W13A5VGWOMCYglDmOMMQGxxGGMMSYgDeIchzHGNHQlJSXk5eVRWFj4nXVxcXGkpqYSHR3t13tZ4jDGmAYgLy+PpKQk0tLSEJFvl6sqBw4cIC8vj06dOvn1XnaoyhhjGoDCwkJatGhxStIAEBFatGhR5UjkdCxxGGNMA1E5aVS3/HTsUJWpU8rLlZmLt3PoeLGzQISKH3kREISK3wFxlznrTv5iVLQ7+bzq5SfbV97Hyffz3b6qWM60D05ZXmlbv2Opeh9Uufz0+zhjLFW8F1Us79IqkZaJsZj6zxKHqVP+uWY3v3p/g9dhmCo0jY/mrXGD6NYmyetQTIhZ4jB1RmlZOc8vyOGcNknMfXQwERHOX7yqSkU9Mq147W6jChWvfGuWVV5+sv2p2/Lt+353Hyefn2zDKW1O3faUfVSKxd998J02VWwbQJ8D2sdp/l9RKCwt48fvrmXU9KW884NBpLVMwIQfVa3ysFSgBf1CmjhE5ErgBSASmKaqT1daL+76q4DjwN2qutJd9xhwP84o+BVVfd5d/gfgWqAY2AKMVdVDoeyHCQ/vfrmLbfuPMfWuvt8mDXAPrZzyuxDY8VoTHK/dG8+tU5dw57SlvDN+EO2aNvI6JOMjLi6OAwcOfOcEecVVVXFxcX6/V8hOjotIJPAiMALoDtwuIt0rNRsBpLuPccAUd9ueOEmjP3A+cI2IpLvbzAd6qmovIBv4caj6YMJHcWk5LyzIoVdqE4Z3b+11OKYK6a2TmH1Pf46cKGHUtKXsP1rkdUjGR2pqKgUFBWzatImNGzd++9i0aRMFBQWkpqb6/V6hHHH0B3JVdSuAiLwJjAR8D1CPBGarM05aIiJNRaQtcC6wRFWPu9t+AlwP/F5V5/lsvwS4KYR9MGHi7ayd7Dp0gv+9vmfAV4CY2tMzpQmvju3HXdOXcdf0Zbx5/0CaxPt3U5kJrejoaL/v06hOKC/HTQF2+rzOc5f502YdMEREWohIPM6hrPZV7OMe4IOgRWzCUmFJGZMX5dK3YzMuyfjODM8mzGSmNeflu/qyZd9R7p65jGNFpV6HZIIslImjqj8LK5+BqbKNqm4EfodzWOrfwGrglJ8+Efmpu+z1KncuMk5EskQkKz8/P9DYTRj5y9IdfH2kkCeHZ9hoo44YkpHMpNsvYE3eYcb9OYvCkjKvQzJBFMrEkcepo4RUYLe/bVR1uqr2UdUhwEEgp6KRiIwBrgHu1NNcDqCqU1U1U1Uzk5Ptr9S66kRxGX/6eAuDOrfgwq4tvQ7HBODKnm34w029+CL3AA//5UtKysq9DskESSgTx3IgXUQ6iUgMcBswp1KbOcBocQwEDqvqHgARaeX+2wG4AXjDfX0l8CPg+xXnQEz9Nfs/29l/tIgnr8jwOhRTAzf0SeXXI3uwYONefvjOasrKA7vs04SnkJ0cV9VSEXkY+BDnctwZqrpeRMa7618C5uKcv8jFuRx3rM9b/E1EWgAlwEOq+o27fDIQC8x3D1ssUdXxoeqH8c7RolJe+mQLl2Qkk5nW3OtwTA3dNSiNo0Vl/O7fm4iPieK3doFDnRfS+zhUdS5OcvBd9pLPcwUeOs22g0+zvGswYzTh69XPt/HN8RKeGG6jjbrugUu7cLSohBc/2kJibCQ/uepcSx51mN05bsLS4eMlTP1sK8PObc357Zt6HY4Jgh9e0Y2jhaW88tk2kuKieXRoevUbmbBkicOEpWmfb6WgsNRGG/WIiPCLa3tQUFTKc/OzSYyN4p6Lg3NfgaldljhM2Dl4rJgZn2/j6vPa0r1dY6/DMUEUESH8/sZeHC8q41fvbyAxNopb+lV1i5YJZ1aPw4Sdlz/ZwomSMiYMt0MZ9VFUZAQv3N6bIRnJPPXuGt5fU/kqfRPuLHGYsLKvoJBZ/9nOyN4pdG1l03PXV7FRkbw8qi+ZHZvz+Jur+GjTPq9DMgGwxGHCyp8+2kJJmfKYnTit9xrFRDLt7kzObduY8a+t4D9bDngdkvGTJQ4TNvYcPsFflu7gpj6pVs+hgWgcF82se/rToXk8981azqqdViGhLrDEYcLG5EW5KMojQ+1WnYakeUIMr903gBaJsYyZsYxNXx/xOiRTDUscJizsPHict5bv5LZ+HUhtFu91OKaWtW4cx+v3DSAuOoJR05axbf8xr0MyZ2CJw4SFSQtziIgQHrrMRhsNVfvm8bx+3wDKVRk1bSm7D53wOiRzGpY4jOe25h/lbyvzGDWgI22a+F++0tQ/XVu5VQQLnSqC+QVWRTAcWeIwnnthYQ6xUZE8cGkXr0MxYaBnShNevbsfew4Xctf0pRw+XuJ1SKYSSxzGU9l7C5izejdjLkwjOSnW63BMmMhMa87U0X3Zmn/MqgiGIUscxlMT52eTEBPFD4Z09joUE2YGp5+sInj/bKsiGE4scRjPrN99mA/Wfc09F3eiWUKM1+GYMHRlzzY8c3MvFm85wMN/WWlVBMOEJQ7jmYnzs2kcF8W9NkOqOYPrL0jl19f1ZMHGfTz5tlURDAc2O67xxJc7vmHBxn381/e60aRRtNfhmDB318COHCsq5ekPNpEQG8lvrz/PCkF5yBKH8cRz87NpnhDD3RemeR2KqSPGX9KFo4WlTP4ol4SYKH56tVUR9IolDlPrlm07yGc5+/nJVeeQEGs/gsZ/T16RwdGiUqZ97lQRfGyYTYbpBfutNbVKVXl23maSk2K5a2Ca1+GYOkZE+Pk13SkoLGXigmwS7RyZJyxxmFq1eMsBlm47yP9c251GMZFeh2PqoIgI4Xc3nsfx4lJ+/f4GEmMjubVfB6/DalDsqipTa1SVZ+Ztpl2TOG4fYL/opuaiIiN4/rbeXJKRzFPvruWfq62KYG0KaeIQkStFZLOI5IrIU1WsFxGZ5K5fIyJ9fNY9JiLrRGS9iDzus7y5iMwXkRz332ah7IMJno835/PljkM8fHk6sVE22jBnJzYqkpdG9aVfx+ZMeGsVizbt9TqkBiNkiUNEIoEXgRFAd+B2EeleqdkIIN19jAOmuNv2BO4H+gPnA9eISMVZsKeAhaqaDix0X5swp6o8O38z7Zs34ubMVK/DMfVEo5hIpn9bRXAli7fs9zqkBiGUI47+QK6qblXVYuBNYGSlNiOB2epYAjQVkbbAucASVT2uqqXAJ8D1PtvMcp/PAq4LYR9MkHy4fi/rdh3hsaEZREfaEVITPEluFcGOzeO5f1YWX+74xuuQ6r1Q/ganADt9Xue5y/xpsw4YIiItRCQeuApo77Zprap7ANx/W1W1cxEZJyJZIpKVn59/1p0xNVderkycn03nlglc17ud1+GYesi3iuDdry5n4x6rIhhKoUwcVd2ZU3mugCrbqOpG4HfAfODfwGogoOkxVXWqqmaqamZycnIgm5oge3/tHjbvLeDx4RlE2WjDhEhFFcFG0ZHcNd2qCIZSKH+L8zg5SgBIBSpf+nDaNqo6XVX7qOoQ4CCQ47bZ6x7Owv13XwhiN0FSWlbO8wuy6dY6iWvOa+t1OKaea988ntd8qgjusiqCIRHKxLEcSBeRTiISA9wGzKnUZg4w2r26aiBwuOIwlIi0cv/tANwAvOGzzRj3+RjgHyHsgzlLf1+1m635x5gwPJ2ICJsewoRe11aJVkUwxEKWONyT2g8DHwIbgbdVdb2IjBeR8W6zucBWIBd4BXjQ5y3+JiIbgH8CD6lqxRmvp4HhIpIDDHdfmzBUUlbOpIU59GjXmO/1aON1OKYB6ZnShJlj+/G1W0Xw0PFir0OqV0S1/k9RnJmZqVlZWV6H0eC8sWwHP353LTPuzuTyc1p7HY5pgD7LyefemVl0b9eY1+4bQKLNjRYQEVmhqpmVl9uZShMSRaVl/HFhDhd0aMpl3aq88M2YkBucnswf77iAtbsOc/8sqyIYLJY4TEi8uWwnuw8X8uTwbjb1tfHU93o4VQSXbDvAQ69bFcFgsMRhgu5EcRmTP8qlf6fmXNS1hdfhGONUERzZk4Wb9vGEVRE8a3bAzwTda0u+Ir+giMm3X2CjDRM2RrlVBP/vg00kxETyfzdYFcGassRhgupYUSlTPtnC4PSWDOhsow0TXn5wSRcKKqoIxkbxM6siWCOWOExQzVy8nYPHinlieIbXoRhTpYoqgtM/30ZSXBSPD7Of1UBZ4jBBc6SwhKmfbmXoOa24oIPNdm/CU0UVwaNFpTy/IIfE2CjuG9zZ67DqFEscJmimf7aNwydKmGCjDRPmIiKEp29wqgj+5l8bSYyN4rb+VlzMX5Y4TFB8c6yY6Z9vY0TPNvRMaeJ1OMZUKyoygudvvYBjRVn8+L21JMRGce35NnuzP+xyXBMUUz/byrHiUhttmDolJirCqSKY5lQRXLjRqgj6wxKHOWv5BUXM/GI73z+/HRmtk7wOx5iANIqJZPqYTLq3a8wDr1sVQX9Y4jBn7aVPtlBUWsZjQ9Orb2xMGEqKi2bW2P6ktYjnvllZrLQqgmdkicOcla8PF/Lakq+4oU8qnZMTvQ7HmBprlhDDa/cOIDkplrtnLLMqgmdgicOclRc/yqWsXG20YeqFVo3jeO3eAcTHRHHX9GVszT/qdUhhyRKHqbG8b47z5vId3NKvPe2bx3sdjjFBUVFFUK2K4GlZ4jA19seFuYgIj1ze1etQjAmqrq0SmX1vfwqKSrnzlSXsKyj0OqSwYonD1Mj2/cf468o87ujfgbZNGnkdjjFB16OdU0Vw75EiRk9fZlUEfVjiMDUyaWEO0ZHCg5d18ToUY0Kmb8fmvDI6k635xxjz6nKOFpV6HVJYsMRhApa7r4D3Vu1izKA0WiXFeR2OMSF1cXpLJt9xAet2Hea+WcutiiCWOEwNTFyQQ3x0JD+4xEYbpmG4okcbnr35fJZuO8iDVkXQEocJzIbdR/jXmj3cc3EnmifEeB2OMbXmugtS+M11PVm0aR8T3lrVoKsI2iSHJiATF2STFBfFfRfbNNSm4blzgFNF8LdzN5EQE8XTNzbMKoIhHXGIyJUisllEckXkqSrWi4hMctevEZE+PusmiMh6EVknIm+ISJy7vLeILBGRVSKSJSL9Q9kHc9KavEPM37CX+wd3pkl8tNfhGOOJcUO68MjlXXkraye/fn8jqg1v5BGyxCEikcCLwAigO3C7iHSv1GwEkO4+xgFT3G1TgEeBTFXtCUQCt7nb/B74par2Bn7uvja14Nl52TSLj2bsRWleh2KMp54YnsHdF6Yx44ttPL8gx+twal0oD1X1B3JVdSuAiLwJjAQ2+LQZCcxWJ2UvEZGmItLWJ7ZGIlICxAO73eUKNHafN/FZbkIoa/tBPsnO56kR55AUZ6MN07BVVBE8VlTKCwtznMO3DaiKYCgTRwqw0+d1HjDAjzYpqpolIs8AO4ATwDxVnee2eRz40F0fAVxY1c5FZBzOKIYOHayy19l6dl42LRNjGD2oo9ehGBMWIiKEp2/sxfHiMn7zr40kxEZxewOpIhjKcxxVnTGqfDCwyjYi0gxnNNIJaAckiMgod/0DwARVbQ9MAKZXtXNVnaqqmaqamZycXKMOGMfiLfv5z9YDPHhpV+Jj7HoKYypERggTb+3Npd2S+cl7a5mzumEcAAll4sgD2vu8TuW7h5VO12YYsE1V81W1BHiXkyOLMe5rgHdwDomZEFFVnpuXTZvGcdwxoGH8NWVMICqqCPZPa84Tb61iwYb6X0UwlIljOZAuIp1EJAbn5PacSm3mAKPdq6sGAodVdQ/OIaqBIhIvzrVuQ4GN7ja7gUvc55cDDe/MVC36JDufrK++4eHLuxIXHel1OMaEpbjoSKaNyaRHu8Y8+JeVLM6t31UEQ5Y4VLUUeBj4EOdL/21VXS8i40VkvNtsLrAVyAVeAR50t10K/BVYCax145zqbnM/8KyIrAZ+i3sewwSfqvLc/GxSmzXilsz21W9gTAOWFBfNzIoqgrPrdxVBaQjXIGdmZmpWVpbXYdQ58zfs5f7ZWfz+xl7c0s8ShzH+2HekkJtf/g/fHCvmzXGD6N6ucfUbhSkRWaGqmZWX25Qjpkrl5cqz8zaT1iKeG/qkeB2OMXVGRRXBhNgoRs9YWi+rCFriMFX6YN3XbPq6gMeHZRAVaT8mxgTiZBVBGDVtKXnfHPc6pKCybwTzHWXlysQF2aS3SuTa89t5HY4xdVKX5ET+fO8AjhaVMmra0npVRdASh/mOOat3kbvvKBOGZxAZ0fAmcDMmWLq3a8yrY/uzr6CIu6bVnyqCljjMKUrKynlhQQ7ntm3MlT3aeB2OMXVe347NeGV0Jtv2H2PMjGX1ooqgJQ5zindX5rH9wHGeHJ5BhI02jAmKi7q25MU7+7Bu9xHunVn3qwha4jDfKiotY9LCXM5v35Sh57byOhxj6pXh3Vvz3C3ns2y7U0WwuLTuVhG0xGG+9fbynew6dIInh2c0yOI0xoTayN4p/O915zlVBN+uu1UEbcY6A0BhSRmTP8qlX1ozBqe39DocY+qtOwZ04FhRKf87dyMJMZE8fUOvOndY2BKHAeD1pTvYe6SI52+9wEYbxoTY/UM6U1BYwqRFuSTERvHza7rXqd87SxyG48WlTPk4l4u6tmBQlxZeh2NMgzBheAYFRaW8+sV2kuKieWJ4htch+c0Sh2HW4q/Yf7SYl4d38zoUYxoMEeG/r3aqCE5amENSbBT3D6kbVQQtcTRwBYUlvPzpFi7rlkzfjs28DseYBiUiQvi/G3pxrKjMOecRG1Un6t5Y4mjgZny+nUPHS3jCRhvGeKKiiuDx4lJ++ve1JMRGMrJ3eE8sapfjNmCHjhcz7bOtXNG9NeelNvE6HGMarJioCKZUVBF8ezXzw7yKoCWOBuyVz7ZytLiUJ66oOyfljKmv4qIjmX53P3q2a8xDf1nJF2FcRdASRwN14GgRr36xnavPa8s5bepuoRlj6pPE2Chmju1PpxYJ3D87ixVfhWcVQUscDdTLn26lsKSMx4fZaMOYcNIsIYY/39ufVkmxjH11GRt2H/E6pO/wK3GIiN1KXI/sO1LIrMXbue6CFLq2SvQ6HGNMJa0ax/HafSerCG4JsyqCZ0wcInKtiOQDa0UkT0QurKW4TAj96eMtlJYrjw1N9zoUY8xppDaL5/X7BgDhV0WwuhHH/wKDVbUtcCPwf6EPyYTSrkMn+MvSHdySmUrHFgleh2OMOYPOyYnMvmcAx4pKuXPaUvYdCY8qgtUljlJV3QSgqkuBpNCHZEJp8qJcAB6+3EYbxtQF3ds1ZuY9/ckvKOKu6cv45pj3VQSrSxytROSJikcVr89IRK4Ukc0ikisiT1WxXkRkkrt+jYj08Vk3QUTWi8g6EXlDROJ81j3ivu96Efl9IB1uyHYcOM47WTu5vX97Upo28jocY4yf+nRwqwgeOMbdry6joLDE03iqSxyv4IwyKh6+r894VlVEIoEXgRFAd+B2EeleqdkIIN19jAOmuNumAI8CmaraE4gEbnPXXQaMBHqpag/gGX86auCFhTlERggPXdbV61CMMQG6qGtL/nSHW0VwVpanVQTPOOWIqv7ydOtEpF81790fyFXVrW77N3G+8Df4tBkJzFZVBZaISFMRaesTWyMRKQHigd3u8geAp1W1yI1xXzVxGGBL/lHe+zKPey7qRKvGcdVvYIwJO8PcKoKPv7WKB15bwct3ZRITVft3VQS0RxHpLiK/EpEc3NHBGaQAO31e57nLqm2jqrtwRhI7gD3AYVWd57bJAAaLyFIR+eR0CUxExolIlohk5efn+9W/+uz5BTnERUcy/tIuXodijDkLI3un8Nvrz+OjzflMeMubKoLVTnIoIh2B291HKdAR5xDS9uo2rWJZ5R5W2UZEmuGMRjoBh4B3RGSUqr7mxtwMGAj0A94Wkc7uqOXkm6hOBaYCZGZm1s36jEGy+esC3l+zmwcu6ULLxFivwzHGnKXb+ztVBH/zr43Ex0Tyuxtrt4pgdfdxLAbmAtHATaraFyjwI2mAM3po7/M6lZOHm6prMwzYpqr5qloCvAtc6LPNu+pYBpQDdoPiGUycn01iTBTj6shc/8aY6t03uDOPDk3nnRV5/Or9DVT62zmkqjtUlY9zIrw1kOwu8ze65UC6iHQSkRick9tzKrWZA4x2r64aiHNIag/OIaqBIhIvTj3FocBGd5u/A5cDiEgGEAOE72xgHlu36zD/Xv819w7uRNP4GK/DMcYE0YRh6dxzUSdmLt7OxPnZtbbf6k6OjxSRJjg3//1SRLoCTUWkv/vX/pm2LRWRh4EPca6KmqGq60VkvLv+JZzRzFVALnAcGOuuWyoifwVW4hwe+xL3sBMwA5ghIuuAYmBM5cNU5qTn5mfTpFE091zcyetQjDFBJiL89zXnOlUEF+WSGBfFuCGhP48pgXznikhr4Fac0UN7VW1fzSZhITMzU7OysrwOo9at+OobbpyymP93ZTcevNQuwTWmviorVx5980v+tWYPv73+vKBVERSRFaqaWXl5QBUAVXUvMAmY5J40N2Fs4vxsWiTEMGZQmtehGGNCKDJCmHhLb04Ul9VKFcEzJg4RqXxOorLvBzEWE0RLth7g89z9/Ozqc0mItQrBxtR3MVER/OnOPtz96jKeeHs1jaIjuaJHm5Dsq7pvlEE491m8ASyl6stnTZhRVZ6bl03rxrGMGmgDQ2MairjoSKaN6ced05by8F++ZMbd/bg4PfgXnVZ3VVUb4CdAT+AFYDiwX1U/UdVPgh6NCYrPc/ezbPtBHrqsK3HRkV6HY4ypRYmxUcwa249OLUNXRfCMiUNVy1T136o6BueGu1zgYxF5JOiRmKBQVZ6Zl01K00bc2q9OXLtgjAmypvEx/Pm+/vTr1JzkENz068+d47HA1Th3jqfhnBx/N+iRmKBYtGkfq3ce4ukbziM2ykYbxjRUrZLimH1P/5C8d3Unx2fhHKb6APilqq4LSRQmKMrLlefmZ9OheTw39k31OhxjTD1V3YjjLuAYzsSCjzo3cQPOSXJV1cYhjM0E6MP1X7N+9xGeu+V8oiNrf8ZMY0zDUN2d4/btU0eUlSsTF2TTJTkhpNdvG2OMJYZ64v01u8nee5QJwzOIrMVZMo0xDY8ljnqgtKyc5xfkcE6bJK7q2bb6DYwx5ixY4qgH3vtyF9v2H2PC8IxanZPfGNMwWeKo44pLy3lhYQ7npTThiu6tvQ7HGNMAWOKo495ZsZO8b07wxBUZ+Fz1ZowxIWOJow4rLClj8qJc+nZsxqUZydVvYIwxQWCJow57Y9kO9hwu5MnhNtowxtQeSxx11IniMl78aAsDOzfnwq5Wct0YU3sscdRRs/+znf1Hi3jyim5eh2KMaWAscdRBR4tKeemTLQzJSKZfWnOvwzHGNDCWOOqgmV9s45vjJTw5PMPrUIwxDZAljjrm8IkSpn66lWHntub89k29DscY0wCFNHGIyJUisllEckXkqSrWi4hMctevEZE+PusmiMh6EVknIm+ISFylbX8oIioiDerM8PTPtnKksJQnbLRhjPFIyBKHiEQCLwIjgO7A7SLSvVKzEUC6+xgHTHG3TQEeBTJVtScQCdzm897tccrY7ghV/OHo4LFipn++javPa0v3djajvTHGG6EccfQHclV1q6oWA28CIyu1GQnMVscSoKmIVMzSFwU0EpEoIB7Y7bPdROD/ARrC+MPOy59u4XhJGY8PS/c6FGNMAxbKxJEC7PR5necuq7aNqu4CnsEZUewBDqvqPAAR+T6wS1VXn2nnIjJORLJEJCs/P//sehIG9hUUMmvxdq7rnUJ66ySvwzHGNGChTBxV3cpceYRQZRsRaYYzGukEtAMSRGSUiMQDPwV+Xt3OVXWqqmaqamZyct2fjmPKx1soKVMeG2qjDWOMt0KZOPKA9j6vUzn1cNOZ2gwDtqlqvqqWAO8CFwJdcJLJahHZ7rZfKSJtQtKDMLHn8AleX7qDG/ukkNYywetwjDENXCgTx3IgXUQ6iUgMzsntOZXazAFGu1dXDcQ5JLUH5xDVQBGJF2cSpqHARlVdq6qtVDVNVdNwEk8fVf06hP3w3ORFuagqj1xuow1jjPfOWHP8bKhqqYg8DHyIc1XUDFVdLyLj3fUvAXOBq4Bc4Dgw1l23VET+CqwESoEvgamhijWc7Tx4nLezdnJrv/a0bx7vdTjGGBO6xAGgqnNxkoPvspd8nivw0Gm2/QXwi2reP+3sowxvf1yUg4jw8GU22jDGhAe7czyMbdt/jL+t3MWoAR1p0ySu+g2MMaYWWOIIYy8syCYmMoIHLu3idSjGGPMtSxxhKmdvAf9YvZsxF6aRnBTrdTjGGPMtSxxhauKCbBJiovjBkM5eh2KMMaewxBGG1u8+zNy1X3PPxZ1olhDjdTjGGHMKSxxhaOL8bBrHRXHvxZ28DsUYY77DEkeYWbXzEAs27mPckM40aRTtdTjGGPMdljjCzHPzs2meEMPdF9lowxgTnixxhJHl2w/yaXY+4y/pTGJsSO/NNMaYGrPEESZUlWc+3ExyUix3DUzzOhxjjDktSxxhYvGWAyzddpCHLu1Co5hIr8MxxpjTssQRBlSVZ+dtpm2TOG7r38HrcIwx5owscYSBj7PzWbnjEI9cnk5ctI02jDHhzRKHx1SV5+Zl0755I27OTPU6HGOMqZYlDo/N27CXtbsO8+jl6URH2sdhjAl/9k3lofJyZ7TRuVf7JwoAAA/nSURBVGUC11+Q4nU4xhjjF0scHvrX2j1s3lvAY8PSibLRhjGmjrBvK4+UlpUzcUE23VoncW2vdl6HY4wxfrPE4ZF/rNrN1vxjTBieTkSEeB2OMcb4zRKHB0rKynlhYQ492jXmez3aeB2OMcYExBKHB/62Io8dB4/z5BUZiNhowxhTt1jiqGVFpWVMWphD7/ZNuaxbK6/DMcaYgIU0cYjIlSKyWURyReSpKtaLiExy168RkT4+6yaIyHoRWScib4hInLv8DyKyyW3/nog0DWUfgu2t5TvZfbiQH17RzUYbxpg6KWSJQ0QigReBEUB34HYR6V6p2Qgg3X2MA6a426YAjwKZqtoTiARuc7eZD/RU1V5ANvDjUPUh2ApLypi8KJf+nZpzUdcWXodjjDE1EsoRR38gV1W3qmox8CYwslKbkcBsdSwBmopIW3ddFNBIRKKAeGA3gKrOU9VSt80SoM7M0/Hakq/YV1DEk8Pt3IYxpu4KZeJIAXb6vM5zl1XbRlV3Ac8AO4A9wGFVnVfFPu4BPqhq5yIyTkSyRCQrPz+/hl0InmNFpUz5eAuD01syoLONNowxdVcoE0dVf1KrP21EpBnOaKQT0A5IEJFRp2wo8lOgFHi9qp2r6lRVzVTVzOTk5ICDD7aZi7dz4FgxTwzP8DoUY4w5K6FMHHlAe5/XqbiHm/xoMwzYpqr5qloCvAtcWNFIRMYA1wB3qmrlZBR2jhSWMPXTrQw9pxUXdGjmdTjGGHNWQpk4lgPpItJJRGJwTm7PqdRmDjDavbpqIM4hqT04h6gGiki8OCcDhgIbwblSC/gR8H1VPR7C+INm+mfbOHyihAk22jDG1ANRoXpjVS0VkYeBD3GuipqhqutFZLy7/iVgLnAVkAscB8a665aKyF+BlTiHo74EprpvPRmIBea7J5iXqOr4UPXjbB06XsyMz7dxZY829Exp4nU4xhhz1kKWOABUdS5OcvBd9pLPcwUeOs22vwB+UcXyrkEOM6SmfrqVo8WlNtowxtQbdud4CO0/WsSrX2zn2l7t6NYmyetwjDEmKCxxhNBLH2+hqLSMx4elex2KMcYEjSWOENl7pJA/L/mKG/qk0jk50etwjDEmaCxxhMiLH+VSVq48NtRGG8aY+sUSRwjkfXOcN5bt4JZ+7WnfPN7rcIwxJqgscYTA5EW5CMLDl9WpC8CMMcYvljiCbPv+Y7yzIo87BnSgXdNGXodjjDFBZ4kjyCYtzCE6Unjwsi5eh2KMMSFhiSOIcvcd5e+rdjF6UBqtkuK8DscYY0LCEkcQPb8gm0bRkfxgSGevQzHGmJCxxBEkG/cc4f01exh7USdaJMZ6HY4xxoSMJY4gmTg/m6S4KO4fbKMNY0z9ZokjCNbmHWbehr3cP7gzTeKjvQ7HGGNCyhJHEDw7fzNN46MZe1Ga16EYY0zIWeI4Syu+OsjHm/MZf0kXkuJstGGMqf8scZylZ+dl0zIxhtGDOnodijHG1ApLHGfhP1sOsHjLAR68tCvxMSGtiWWMMWHDEkcNqSrPzd9Mm8Zx3DGgg9fhGGNMrbHEUUOf5uxn+fZveOjyrsRFR3odjjHG1BpLHDWgqjw3bzMpTRtxa2Z7r8MxxphaZYmjBhZs3MfqvMM8NjSdmCj7LzTGNCz2rReg8nLlufnZpLWI54Y+KV6HY4wxtS6kiUNErhSRzSKSKyJPVbFeRGSSu36NiPTxWTdBRNaLyDoReUNE4tzlzUVkvojkuP82C2UfKvv3+q/ZuOcIjw/LICrS8q4xpuEJ2TefiEQCLwIjgO7A7SLSvVKzEUC6+xgHTHG3TQEeBTJVtScQCdzmbvMUsFBV04GF7utaUeaONtJbJXLt+e1qa7fGGBNWQvknc38gV1W3qmox8CYwslKbkcBsdSwBmopIW3ddFNBIRKKAeGC3zzaz3OezgOtC2IdT/HP1bnL3HWXC8AwiI6S2dmuMMWEllIkjBdjp8zrPXVZtG1XdBTwD7AD2AIdVdZ7bprWq7gFw/21V1c5FZJyIZIlIVn5+/ll3prSsnOcXZHNu28Zc2aPNWb+fMcbUVaFMHFX9Sa7+tHHPW4wEOgHtgAQRGRXIzlV1qqpmqmpmcnJyIJtW6d2Vu9h+4DhPDM8gwkYbxpgGLJSJIw/wvckhlZOHm6prMwzYpqr5qloCvAtc6LbZW3E4y/13XwhiP0VxaTkvLMzh/NQmDDu3ygGOMcY0GKFMHMuBdBHpJCIxOCe351RqMwcY7V5dNRDnkNQenENUA0UkXkQEGAps9NlmjPt8DPCPEPYBgLeydrLr0AmeuKIbTjjGGNNwhWxmPlUtFZGHgQ9xroqaoarrRWS8u/4lYC5wFZALHAfGuuuWishfgZVAKfAlMNV966eBt0XkXpwEc3Oo+gBQWFLG5EU59EtrxpD0lqHclTHG1AkhndJVVefiJAffZS/5PFfgodNs+wvgF1UsP4AzAqkVry/dwd4jRTx/6wU22jDGGOzO8TM6XlzKlI9zubBLCwZ1aeF1OMYYExYscZzB7P98xf6jxTx5RYbXoRhjTNiwxHEGLRNjuSUzlb4dm3sdijHGhA0rW3cGN/VN5aa+qV6HYYwxYcVGHMYYYwJiicMYY0xALHEYY4wJiCUOY4wxAbHEYYwxJiCWOIwxxgTEEocxxpiAWOIwxhgTEHHmGazfRCQf+KqGm7cE9gcxHC9ZX8JPfekHWF/C1dn0paOqfqcSXoNIHGdDRLJUNdPrOILB+hJ+6ks/wPoSrkLRFztUZYwxJiCWOIwxxgTEEkf1plbfpM6wvoSf+tIPsL6Eq6D3xc5xGGOMCYiNOIwxxgTEEocxxpiAWOJwiciVIrJZRHJF5Kkq1o8UkTUiskpEskTkYi/i9Ed1ffFp109EykTkptqMz19+fCaXishh9zNZJSI/9yJOf/jzmbj9WSUi60Xkk9qO0V9+fC7/5fOZrHN/xsKujKYf/WgiIv8UkdXuZzLWizj94UdfmonIe+532DIR6XlWO1TVBv8AIoEtQGcgBlgNdK/UJpGT54R6AZu8jrumffFptwiYC9zkddw1/EwuBd73OtYg9aUpsAHo4L5u5XXcZ/Pz5dP+WmCR13HX8DP5CfA793kycBCI8Tr2GvblD8Av3OfnAAvPZp824nD0B3JVdauqFgNvAiN9G6jqUXX/14EEIFyvKqi2L65HgL8B+2ozuAD424+6wJ++3AG8q6o7AFS1vnwutwNv1EpkgfGnHwokiYjg/OF4ECit3TD94k9fugMLAVR1E5AmIq1rukNLHI4UYKfP6zx32SlE5HoR2QT8C7inlmILVLV9EZEU4HrgpVqMK1B+fSbAIPdQwgci0qN2QguYP33JAJqJyMciskJERtdadIHx93NBROKBK3H+QAk3/vRjMnAusBtYCzymquW1E15A/OnLauAGABHpD3QEUmu6Q0scDqli2XdGFKr6nqqeA1wH/DrkUdWMP315HviRqpbVQjw15U8/VuLMpXM+8Efg7yGPqmb86UsU0Be4Gvge8N8ikhHqwGrAr98V17XAF6p6MITx1JQ//fgesApoB/QGJotI41AHVgP+9OVpnD9MVuEcbfiSsxg9RdV0w3omD2jv8zoV56+MKqnqpyLSRURaqmq4TYTmT18ygTedETgtgatEpFRVw+mLt9p+qOoRn+dzReRPdfgzyQP2q+ox4JiIfAqcD2TXToh+C+R35TbC8zAV+NePscDT7iHqXBHZhnN+YFnthOg3f39XxgK4h962uY+a8frETjg8cBLoVqATJ08u9ajUpisnT473AXZVvA6nhz99qdR+JuF5ctyfz6SNz2fSH9hRVz8TnEMiC9228cA6oKfXsdf05wtognNOIMHrmM/iM5kC/I/7vLX7O9/S69hr2JemuCf2gfuB2WezTxtxAKpaKiIPAx/iXKEwQ1XXi8h4d/1LwI3AaBEpAU4At6r7KYQTP/sS9vzsx03AAyJSivOZ3FZXPxNV3Sgi/wbWAOXANFVd513UVQvg5+t6YJ46I6iw42c/fg3MFJG1OIeDfqThN5r1ty/nArNFpAzn6r17z2afNuWIMcaYgNjJcWOMMQGxxGGMMSYgljiMMcYExBKHMcaYgFjiMMYYExBLHKbeEZGfurOZVsxmPKCa9jNrMkOwiKSJyB012K7K/YnIOW68X4pIlxq87+PuNB/GhJQlDlOviMgg4Bqgj6r2AoZx6jw+wZSGMzlhsFwH/ENVL1DVLTXY/nGcmwf9JiJ2L5cJmCUOU9+0xZm6owhAVfer6m4AEekrIp+4kwh+KCJtK298ujYi0lVEFrgTKq50RwRPA4PdUcIEEYkUkT+IyHJ3tPMDd1sRkckiskFE/gW0qmK/V+F88d8nIh+5y0a5tRNWicjLIhLpLp8iTk2Y9SLyS3fZozhzKn3ks/1Rn/e/SURmus9nishzbrvfudPn/Nvt82cico7b7mZx6mmsdqdAMcbh9e3y9rBHMB8401+vwpnj6U/AJe7yaGAxkOy+vhXnDltwp12pps1S4Hr3eRzOX/aX4lMPBBgH/Mx9Hgtk4UwDcQMwH+eu3nbAIaqY5gX4H+CH7vNzgX8C0e7rPwGj3efN3X8jgY+BXu7r7fhMiQEc9Xl+EzDTp7/vA5Hu64VAuvt8AG79DJwZYVPc5029/mztET4PG6aaekVVj4pIX2AwcBnwljgV0bKAnsB8d3LHSGBPpc27VdVGRJJwvkDfc/dRCOC28XUF0Mvn/EUTIB0YAryhzmzEu0VkkR9dGYozW+5ydz+NOFk75RYRGYczR1FbnFoLa/x4T1/vqGqZiCQCFwLv+PQn1v33C5wpN94G3g3w/U09ZonD1DvuF/THwMfuPENjgBXAelUddIZNpao2AUylLcAjqvphpe2vIvDCXwLMUtUfV3qvTsAPgX6q+o17+CnuNO/hu8/KbSrmkIoADqlq7+9srDrevbDgamCViPRW1QMB9sPUQ3aOw9QrItJNRNJ9FvUGvgI2A8nuyXNEJFq+W/ipyjbqTEmdJyLXuctj3auXCoAkn+0/xJl0MdptlyEiCcCnwG3uOZC2OCOh6iwEbhKRVu57NReRjkBjnC/9w+JUcBvhs03lePaKyLkiEoEz6eB3uH3bJiI3u/sRETnffd5FVZeq6s+B/Zw6dbdpwGzEYeqbROCPItIUp1BNLjBOVYvdQ0iTRKQJzs/+88D6ig2raXMX8LKI/AooAW7GOTxUKiKrcc4bvIBzpdVKcY775ONcKfUecDnOOYNs4JPqOqGqG0TkZ8A894u/BHhIVZeIyJduTFtxDidVmAp8ICJ7VPUy4Cmccxk7caZpTzzN7u4Eprj7i8YpPboa+IObhAUnka2uLm7TMNjsuMYYYwJih6qMMcYExBKHMcaYgFjiMMYYExBLHMYYYwJiicMYY0xALHEYY4wJiCUOY4wxAfn/FIC1p97uXKQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(selection_quota_list, map_list)\n",
    "pyplot.ylabel('MAP')\n",
    "pyplot.xlabel('Selected features')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
